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The apparatus and 
method of the invention 
relate to data entry in 
ideographic languages, 
including Chinese, 
Japanese and Korean, and 
to interactive displays 
and interactive display 
methods for use by persons 
temporarily or permanently 
lacking normal motor 
capabilities. In a preferred 
embodiment, each option 
of a menu is associated 
respectively with a 
selectable region (3606), 
(3610), etc.) displayed 
adjacent an edge of a 
display (2112), forming 
a perimeter menu and 
leaving a region (3680) in 
the center of the perimeter 
menu for the output of 
an application program. 
Selectable regions may be 
on the display, outside the 

display, or both. A menu option may be selected by clicking on the associated selectable region, by dwelling on it for a selection threshold 
period or by a cursor path toward the selectable region, or by a combination thereof. Remaining dwell time required to select a selectable 
region is preferably indicated by the brightness of the selectable region. Submenus of a perimeter menu may also be perimeter menus and 
the location of a submenu option may be foretold by the appearance of its parent menu option. Menu options may be ideographs sharing 
a sound, a structure or another characteristic. Ideographs, which may be homophones of one another, may be associated with colored 
indicating regions and selection of an ideograph may be made by speaking the name of the associated color. 
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Method of and Apparatus for Data Entry 
Technical Field 

5 The present invention relates generally to interactive display terminals and interactive display 
methods, and more to interactive display terminals and interactive display methods for use by 
persons temporarily or permanently lacking normal motor capabilities. It also relates to systems 
and methods for the assessment of the motor capabilities of persons lacking normal motor 
capabilities. It further relates to interactive display terminals and interactive display methods for 

10 use in speech synthesis for persons having impaired speech. It also relates to systems and methods 
for the control of devices, including appliances, by persons lacking normal motor capabilities. It 
further relates to interactive display terminals and interactive display methods for selecting one 
menu option from a menu. It further relates to systems and methods utilizing sound recognition for 
selecting a menu option from a menu. It further relates to data and order entry systems including, 

15 and data and order entry methods utilizing, an interactive display terminal. It also relates to 
interactive display terminals and interactive display methods for displaying and selecting 
ideographic characters, such as are used in the Chinese, Japanese and Korean languages. It also 
relates to interactive display terminals and interactive display methods for producing an indication 
of progress toward and/or away from selection of a menu option. 

20 

A portion of the disclosure of this patent document contains material which is subject to copyright 
protection. The copyright owner has no objection to the facsimile reproduction by anyone of the 
patent document or the patent disclosure, as it appears in the World Intellectual Property 
Organization patent file or records, but otherwise reserves all copyright rights whatsoever. 

25 

Background Art 

Many persons suffer from various neurogenic muscular disorders, such as Cerebral Palsy ("CP"), 
30 Traumatic Brain Injury, Spinal Cord Injury, Muscular Dystrophy, Amyotrophic Lateral Sclerosis 
and Multiple Sclerosis. These conditions can result in a reduced ability to voluntarily control or 
prevent the movement of parts of the body, including the head, limbs and digits, muscle stiffness, 
weakness, limited range of motion, abnormal posture, involuntary muscle tremors, involuntary 
muscle activity causing involuntary motion, impaired ability to voluntarily stop motion, impaired 
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ability to coordinate muscle activity, and/or impaired ability to sense the position of a part of tli 
body. Any one of these symptoms may impair an affected individual's fine motor control. 
Moreover, while some individuals affected by a neuromuscular disorder may be able to exercise 
fine motor control with enormous effort, the struggle to do so often fatigues the individual, limiting 
5 the period of time the individual is capable or comfortable performing the fine motor control tasl' . 

Neuromuscular disorders are often systemic in effect, impairing an individual's ability to operate 
prosthetic devices, such as a wheelchair, and to perform the activities of daily life, such as 
speaking, walking and operating household appliances. Speech is frequently affected since the 

10 mechanics of producing speech require coordination of many muscle groups ~ the muscles of the 
diaphragm which push air over the vocal cords, the muscles of the larynx, jaws, tongue and lips. 
The inability to use or coordinate these muscle groups may result in impaired speech. Depending 
upon the degree of impairment, speech may be totally absent, present but impaired to the point of 
unintelligibility. or intelligible on the whole but with occasional unintelligible words. The ability 

15 to walk is often affected since walking requires coordination and voluntary control of many muscle 
groups. Furthermore, impaired fine motor control may prevent or impede an individuals from 
effectively operating household appliances or computer input devices. 

Devices are available which produce speech, control appliances and facilitate computer access for 
20 some persons having neuromuscular disorders ("NMD operators"). Devices which produce speech 
for individuals whose own speech is impaired, called Augmentative and Alternative 
Communication ("AAC") devices, allow the operator to select words or phrases by spelling the 
words, by specifying an abbreviation for the phrase or by selecting a sequence of symbols, and then 
speak the selected words or phrases using an electronic speech synthesizer. However, due to the 
25 systemic nature of neuromuscular disorders, NMD operators are often unable to efficiently use a 
standard keyboard and mouse. For example, an NMD operator who is unable to stop the 
movement of a limb with precision, when attempting to use a keyboard or mouse, may move his 
arm toward the target key or move the cursor toward the target object on the display but overshoot 
the target. If he has involuntary tremors and cannot hold a limb still, then, when attempting to use 
30 a keyboard, he may hit keys adjacent to his target key. If he has involuntary motion moving left to 
right, then, when attempting to use a keyboard, he may have difficulty accessing an intended key 
on the right side of the keyboard. 

The benefits of interfacing an NMD operator to a general purpose computer so that he may contrc/ 
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the computer and devices attached to it ("computer access") are both numerous, because many of 
the problems faced by the disabled are susceptible to a computer-driven solution, and profound, 
because of the psychological deprivation occasioned by a severe physical disability. The benefits 
potentially obtained through computer access for individuals affected by neuromuscular disorders 
S include: 

a. Speech synthesis. A computer connected to a speech synthesizer enables an NMD operator 
with impaired speech to direct the computer to speak for him. 

b. Device control. A user who is physically unable to operate a household appliance, for 
example, a television, video cassette recorder, compact disc player, radio, alarm clock. 

10 telephone, light, thermostat, dimmer or power switch, may be able to control the appliance 

via a computer equipped with an interface he can control. 

c. Access to general purpose computer applications. NMD operators may make use of the same 
general purpose computer application programs ("applications") as able-bodied users, 
including applications for word processing, database, computer-aided instruction, access to 

15 literature accessible via computer, spreadsheet, time management and computer utilities. 

d. Enhanced self-esteem and peer approval. Adolescents with CP are obviously different from 
their peers. They are often surrounded by non-normative assistive technology, e.g. 
wheelchair or walker, special school bus equipped with a chair lift, stair lift, standing aid, 
AAC device, feeding apparatus, bath seat, toiletting apparatus, etc. In addition, they may 

20 drool, lacking the ability to coordinate lip closure with swallowing. Nonetheless they are 

adolescents and need peer approval to support them in their maturation from dependent 
children to independent adults. Today, demonstrated facility with a computer is an emblem 
of intelligence among adolescents, so computer use provides adolescents the opportunity to 
prove their intelligence and thus potentially rewards NMD operators with both self-esteem 

25 and peer approval. 

e. Privacy. Some severely disabled school-aged children require nearly constant physical 
assistance to transfer them to and from bed, to feed them, help them with toiletting and 
personal hygiene, etc. Because they are constantly attended, all their mistakes in class or 
when doing homework are known to their attendant, often a family member. They do not 

30 have the opportunity to make mistakes in private. Computer use, if it can be done without 

assistance, affords the NMD operator the opportunity to avoid the embarrassment of showing 
their failings to their attendant. 

f. Expanded personal interaction. Some severely disabled individuals, e.g. quadriplegics, are 
essentially incarcerated by their disability. They interact with their family or their caretakers, 
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depending upon whether they live at home or in an institution. Their circle of friends is oft 
very small. Using a computer and a modem, they can expand their circle of friends to 
include the tens of thousands of people who periodically connect to worldwide electronic 
networks to trade information on topics of mutual interest. Moreover, the interaction via 

5 present computer networks is mostly textual; there is no voice or visual interaction between 

users. Since messages are customarily composed and read off-line to minimize connect time 
charges, no one even knows how long it took the sender to enter the text. Electronic 
networks thus afford the disabled user an opportunity to relate to others on an equal footing, 
not as a disabled person to his able-bodied peers, something many NMD operators dearly 

10 want to do but were never able to 



NMD operators vary widely in their motor capabilities. Even individuals having the same medical 
diagnosis may require completely different technologies for computer access. Many NMD 
operators are able to use an oversize keyboard, a devices having a pressure-sensitive surface 

15 divided into squares, each square associated with a letter of the alphabet. The squares may be sized 
to match the operator's abilities, but typically each square is two inches on either side. NMD 
operators who are unable to efficiently use an oversize keyboard may use another conventional 
computer access solution, called an "on-screen keyboard", which, as illustrated in Figure 1, is a 
picture of keyboard drawn on a computer display (1101). The operator selects a letter by pointing 

20 to that letter's key image on the display with a pointing device ("pointer"), then indicating that he 
has reached his target either by operating a switch, a process called selection by click, or by 
maintaining the location indicated by the pointer ("dwelling") on the key image for a 
predetermined period of time (the "selection threshold"), a process called selection by dwell. 
Switch operation includes, but is not limited to, each of the following: opening the switch, closing 

25 the switch, opening the switch multiple times within a predetermined period, and closing the switch 
multiple times within a predetermined period. A program executing on the computer determines 
which letter the operator has selected and processes the letter or passes it to some other application 
program which processes the letter as if it came from a true keyboard. 

30 Conventional pointing devices include a mouse, trackball, joystick (which may be integrated into a 
keyboard, e.g. TrackPoint II®), stylus and graphics tablet, lightpen, thumb wheel, touch screen, 
touch panel, head pointer, occulometer, intraoral pointer and eye tracker. They may be active, e.g. 
a lightpen that emits an infrared beam, or passive, e.g. an eye tracker that uses images of an 
individual's eyes to determine where his eyes are focusing. Conventional switches include a button 
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on the mouse, a switch in the tip of the stylus actuated by pressure or the release of pressure, a 
switch mounted on the user's wheelchair operated by a turn of the head to or the switch below a 
keyboard key. 

5 Dwell time may be continuous or discontinuous depending upon the operator's motor capabilities. 
In continuous dwelling, if the operator moves the cursor from one key image to another region of 
the display, the time accumulated on the key image is discarded so that if the operator returns to 
that key image he must dwell on it for the full selection threshold to select it. Discontinuous 
dwelling, by contrast, compensates for involuntary tremors which pull the operator off the desired 

10 key image. Accumulated dwell time on a key image is remembered, so that on return to a key 
image, the operator need only dwell for a period equal to the difference between the selection 
threshold and the previously accumulated dwell time for that key image. Accumulated dwell time 
is reset to zero for all key images following the selection of any one key image. Conventional on- 
screen keyboards do not indicate to the operator the dwell time associated with any key image. 

15 

There may be a single selection threshold period for all key images or each key image may be 
associated with its own selection threshold period. In the latter case, keys associated with shorter 
selection threshold periods are easier to select than keys associated with longer selection threshold 
periods. 

20 

As was mentioned earlier, computer access permits an NMD operator to run a variety of 
applications. One such application is speech synthesis. In a computer-based speech synthesis 
system, a computer system displaying an on-screen keyboard is connected to a speech synthesizer.. 
The operator spells the desired word or words using the on-screen keyboard. These are then 

25 spoken by the speech synthesizer. Another application of the on-screen keyboard is word 

processing. Figure 2 illustrates an example of a combined display of an on-screen keyboard and a 
word processing application program. The on-screen keyboard (0201) is shown on the lower 
portion of a display connected to a computer system (not shown) which also executes the word 
processing application program whose output (0203) appears on the upper portion of the display. 

30 Letters selected by the operator are input to the word processing application program. 

Due to impaired fine motor control, many NMD operators have difficulty selecting a key image by 
click or by dwell and this difficulty increases as the size of the key image decreases. Figure 1 
shows an on-screen keyboard containing 81 total keys including 26 alphabetic keys, 10 numeric 
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keys, 12 function keys, 4 arrow keys and 29 special purpose keys. Drawing this many key images 
on a display restricts the size of each key image making each very difficult for many NMD 
operators to select. 

5 When a display is shared between application program output (0203) and an on-screen keyboard 
(020 1 ). as is the display shown in Figure 2, the size of each key image must be reduced from its 
size in Figure 1 to allow space for the application program output. Thus, as more display space is 
allotted to application program output, the key images become more difficult for an NMD operator 
to select. 

10 

Many NMD operators have difficulty using the conventional dwell selectable on-screen keyboard 
because they cannot maintain a steady pointer position. The body member with which they control 
the pointer may move slightly ("drift") when they want it to remain still. One approach to this 
problem is a variation of the on-screen keyboard, depicted in Figures 3, 4 and 5 and called a 

15 quaternary on-screen keyboard ("quaternary keyboard"). The quaternary keyboard provides for 

larger key images. The alphabet is divided into four groups of letters, each displayed in one of the 
four quadrants (1302), (1304), (1306) and (1308) of the display, as shown in Figure 3. The 
operator selects one of the four groups by, for example, pointing to and dwelling on one quadrant 
of the display. The selected group is then exploded into four subgroups, each displayed in one 

20 quadrant of the display, as shown in Figure 4. Once more the operator selects one of the four. The 
selected group is exploded into four letters and each letter displayed in one quadrant of the display, 
as shown in Figure 5. The operator again selects one of the four. This letter is then input to an 
application program (not shown). 

25 The quaternary keyboard illustrates the use of a menu hierarchy in computer access. Each of the 
four groups of letters ( 1 302), ( 1 304), ( 1 306) and ( 1 308) is a menu option. Each of these menu 
options is itself a menu which includes other menu options. A menu hierarchy exists if at least one 
of a menu's menu options is itself a menu. Hereinafter, a menu accessed from another menu may 
be called a submenu, and the options of the submenu may be called submenu options. If a menu 

30 hierarchy is narrow and deep, many selections are required to make the desired choice. If a menu 
hierarchy is broad and shallow, each layer is composed of many menu options. 

The quaternary keyboard greatly expands the size of a single key image and thus accommodates 
certain NMD operators with drift or involuntary tremors. The cost of this adjustment is high 
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Instead of selecting a letter with one pointing motion and dwelling for one selection threshold, the 
quaternary keyboard requires three pointing motions and dwelling for three selection thresholds. 
Thus the operator's productivity is dramatically reduced from the standard on-screen keyboard 
depicted in Figure 1 . 

5 

The computer access advantage gained from the quaternary keyboard is greatest when the 
quaternary keyboard occupies the entire display. In this configuration the size of each of the four 
active display regions is maximized, making them easier to hit and dwell on for the operator. 
However, this configuration allows no room on the display for the output of the application 

10 program being run by the operator, the reason he is sitting at the computer in the first place. This 
does not prevent the on-screen keyboard from passing letters to the application program since an 
application program need not be visible to be active, but it does prevent the operator from seeing 
what the application program has to show him. The more of the application program output that is 
displayed, the smaller the on-screen keyboard, the smaller each active region of the on-screen 

15 keyboard and the more difficult access becomes. Figure 6 illustrates a display combining a 
quaternary keyboard and output from two application programs. 

Another conventional structure for selecting of a menu option from a menu is a pie menu A pie 
menu is an opaque region on a display divided into selectable slices, each slice associated with a 
20 menu option. The pie menu suffers some of the drawbacks discussed above, particularly that, while 
displayed the pie menu occupies more space than a linear menu and obscures much of the output of 
the operator's application program. For illustrations and a discussion of pie menus, see Callahan, 
Jack et. al., "An Empirical Comparison of Pie vs. Linear Menus", Computer Science Technical 
Report Series, TR-1919. University of Maryland. September 1987. 

25 

NMD operators who cannot effectively use a conventional keyboard or a pointing device may use a 
computer access method called "joystick patterns". Figure 7 depicts a conventional joystick pattern 
device. The device (1 602) is connected to a joystick and to a computer. The operator pushes the 
joystick to the top, bottom, left, right, top left corner, top right corner, lower left corner or lower 
30 right corner, closing one of eight switch contacts within the joystick housing. That switch position 
is then indicated on the display (1604). A sequence of consecutive of switch closures encodes a 
letter or other programmed output that the device (1602) displays on an LCD display (1606) and 
sends to the connected computer, simulating keyboard input. 
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The conventional joystick pattern device is ill-suited for many NMD operators. The involuntary 
tremors common some neuromuscular disorders may result in unintended switch closures. In 
addition, the device does not provide an indication that the operator is moving a body member in 
an unintended direction until switch closure occurs. For example, an operator with CP who intends 
to move the joystick the right but actually moves it to the upper right receives no indication from 
the device, prior to switch closure, that he's not on target. Moreover, the device requires that the 
operator memorize the encoding of each letter or other output since there's no indication on the 
display (1606) which sequence encodes which letter. Further, the device provides no support for 
head pointing, although the head is often the best controlled part of an NMD operator's body. 

NMD operators who cannot effectively use either a conventional keyboard or a pointing device but 
can reliably actuate a switch may use a computer access method called "scanning", which is 
subdivided by cursor control technique into three types of scanning: automatic, directed and step. 
In automatic scanning all the operators' options, for example, the letters of the alphabet, appear on 
either a static or dynamic display (depending upon the implementation), organized in rows and 
columns. At the scanning interval, usually about one second, a cursor moves from one row to the 
next. When the cursor indicates the row containing the letter the operator wants, he closes a 
switch. The machine now moves the cursor from one letter to the next within the selected row until 
the operator closes the switch again. The operator has now selected one letter. In directed 
scanning, like automatic scanning, the cursor moves at the frequency determined by the scanning 
interval, however, it moves only when the switch is closed. To select an option, such as a row or a 
letter in a row, the operator opens the switch while the cursor indicates the desired option. In step 
scanning, the cursor moves with each switch activation. 

As one can well imagine, writing a sentence via any of these scanning techniques is an extremely 
slow process, since selecting a single letter may take many seconds. 

Problems of computer access cascade and affect the quality of verbal interactions between AAC 
device operators ("AAC operators") and others. People speak much faster than they type. Not 
surprisingly, operators who speak with AAC devices, particularly NMD operators whose motor 
deficits impair their ability to use a keyboard, lag substantially in their conversations. The slow 
pace of an AAC operator's word production disrupts normal verbal interaction. Speaking persons, 
accustomed or not to the AAC operator's slow rate, often lose patience in conversations with AAC 
operators. They may prematurely terminate the conversation, read the AAC device display in an 
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attempt to guess at the AAC operator's intended utterance and so accelerate the interaction, lead thi 
AAC operator, ask predominantly yes/no questions, change the topic of conversation with little 
input from the AAC operator and otherwise dominate the interaction. The AAC operator often has 
difficulty participating as an equal partner in the conversation. He may be unable to change the 
5 topic, interject a humorous comment in a timely fashion or respond to a question before the 

speaking person changes the topic. Slow AAC operators may be perceived as mentally slow. Thus 
the quality of verbal interactions where one party uses an AAC device to speak depends 
significantly upon the AAC operator's rate of word production. 

10 Increasing an operator's letter or menu option selection rate proportionately increases his word 
production rate and increases the operator's productivity in data entry generally. Letter or menu 
option selection time includes the time the operator requires (a) to comprehend the menu options 
displayed, (b) to move the pointer to the desired menu option on the display, and. in selection by 
dwell, (c) the selection threshold period, or, in selection by click, (c) the time required to operate 

15 the switch. Decreasing any one of these increases the operator's productivity, assuming all other 
steps in the selection process are unaffected. 

Personal interactions are composed of more than speech alone. People in conversation gesture to 
one another, use facial expressions, change the object of their gaze and make non-speech utterances 

20 (e.g. "hrnmm-mmmm") to bid for a turn to speak, to grant such a bid made by the other party, to 
request to continue speaking and to acknowledge, accept or dispute what has been said. Ideally, 
the production of speech from an AAC devices does not distract the AAC operator from the 
personal interaction and subject matter of the conversation. This is possible if the operator 
habituates the AAC device access technique and menu structure, producing speech without 

25 focusing on each step of the process, much as automobile drivers habituate mechanical tasks, such 
as changing gears and switching between foot pedals, and focus their attention on pedestrians or 
traffic lights while operating their vehicle. 

Another consequence of personal interaction during conversation for an AAC device operator is 
30 that the operator needs a way to easily enable and disable the AAC device operator interface so that 
movement the operator makes during personal interaction, for example, nodding his head, is not 
interpreted by the AAC device. 

As noted previously, neurogenic muscular disorders may impair the ability of an individual to 
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sense the position of a body member. An NMD operator thus relies more than his able-bodied peer 
on the location of a cursor or similar automated indication of body member position. Conventional 
access methods which use a pointer do not provide additional feedback to the operator of the 
position of a body member. 

5 

Access methods which require the NMD operator to make the same movement for most selections, 
such as single switch access, mouth sticking (the use of a small rod held in the mouth and used to 
depress keys on a keyboard) and head sticking (the use of a rod mounted on the head and used to 
depress keys on a keyboard), may result in repetitive motion injury, particularly after years of use. 

10 

There are several aspects of the invention, each addressing one or more of the problems described 
above and/or one or more problems specifically addressed by that aspect of the invention. The 
objects, disclosure and description of each aspect is separately described below under one of the 
headings: (A) Perimeter Menu, (B) Confinement, (C) Dwell, (D) Path Directness, (E) Intersection, 
15 (F) Alignment, (G) Length Order, (H) Location Indication, (I) Sound Match, and (J) Ideographic 
Languages,. Where there is background art applicable to an aspect in addition to that described 
above, the additional background art is described below. 

A & B. Perimeter Menu and Confinement 

20 

One object of the invention is to facilitate computer access by a disabled operator. 

A further object of the invention is to facilitate menu selection by an operator having impaired 
ability to maintain a body member in a steady position. 

25 

Another object of the invention is to facilitate menu selection by an operator having impaired 
ability to stop motion. 

Yet another object of the invention is to simultaneously display an application program window 
30 and a computer access menu which does not obstruct the application program window. 

Another object of the invention is to allow an operator to enable and disable a menu. 

A still further object of the invention is to synthesize speech for an operator having impaired 
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speech and impaired fine motor control. 

Still another object of the invention is to facilitate device control for a disabled operator. 

5 Another object of the invention is to reduce the cognitive demand of speech synthesis for the 
disabled. 

A still further object of the invention is to enlarge the effective area of a selectable region without 
concomitantly reducing the area available for information display. 

10 

Another object of the invention is to speed data entry. 

Yet another object of the invention is to facilitate computer access for an operator having impaired 
ability to sense the position of a body member used for computer access. 

15 

C. Dwell 

Conventional systems allowing selection by dwell do not provide an indication to the operator of 
either how much dwell time has been accumulated for any selectable region or how much more 

20 dwell time is required to select a selectable region. Consequently, an operator of a conventional 

system who is dwelling on a intended selectable region has no indication, other than his estimation 
from prior use of the system, that he has nearly made his selection and can plan his next movement 
to the next selectable region or that he has very nearly made his selection and can begin moving to 
the next selectable region. Furthermore, an operator of a conventional system who is dwelling on 

25 an unintended selectable region, has no indication, other than his estimation from prior use of the 
system, of how close he is to making an unintended selection and thus how important it is to act 
quickly. Conventional systems using discontinuous dwell give no indication of the accumulated 
dwell time associated with a selectable region either when the operator dwells on that region or 
when the operator ceases dwelling on that region. Some disabled users can dwell relatively easily 

30 on their intended targets for short periods of time, but have difficulty dwelling for long periods. If 
such an operator knows that only a little more dwell time is needed he may be able to satisfy the 
dwell time required for selection, without preparing himself to dwell for an extended period. 

Conventional menu-driven data entry and order entry systems incorporating pointing at intended 

11 
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selections employ a two step selection procedure. In the first step the operator indicates, with a 
pointer, his intended selection. The system then provides feedback, for example, by highlighting 
the indicated selection, showing which selection the operator has indicated. In the second step, the 
operator selects the indicated selection, for example, by operating a switch. Thus, conventional 
5 data entry and order entry systems are ill-suited to circumstances where the operator cannot easily 
operate a switch while maintaining the pointer on the intended selection. 

While the two step procedure is not complicated, many operators require some training to learn it, 
and, if they are infrequent users of the system, these operators may require refresher training. 
10 Simplifying the procedure further would lessen the need for initial and refresher training. 

One object of the invention is to facilitate the use of systems allowing selection by dwell. 

Still another object of the invention is to facilitate device control by the disabled. 

15 

A further object of the invention is to increase the independence of the disabled. 

Yet another object of the invention is to facilitate the use of a data entry or order entry system by 
an intermittent operator. 

20 

Another object of the invention is to facilitate ordering by someone seated in a vehicle. 
D. Path Directness 

25 The on-screen keyboard with dwell selectable key images is ill-suited for use by many NMD 
operators. Selection by dwell may fatigue NMD operators or may require greater fine motor 
control than they bring to this task. Operators with impaired ability to stop motion and those 
having involuntary tremors have difficulty maintaining the location indicated by a pointer on a key 
image for a period sufficient to distinguish intentional dwelling from unintentional dwelling. 

30 Consequently, some NMD operators who try to use on-screen keyboards often miss their target key 
images and/or accidentally select unintended key images. Following such an error, the operator 
must erase his erroneous selection by selecting the backspace or undo key. As the number of 
erroneous selections increases, the operator's productivity decreases markedly, since each error 
requires a correction in which there might be another error. 

12 
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Conventional on-screen keyboards require the ability to select by dwell or by click and thus are 
limited to operators with these capabilities. Conventional on-screen keyboards do not utilize the 
relatively intact motor capabilities of some NMD operators to compensate for impaired ability to 
select by dwell or by click or to speed up the slow process of selecting by dwell. For example, 
5 while an NMD operator may overshoot a key image, his directional control may be relatively 
intact. Conventional on-screen keyboards do not exploit this capability. 

The dominant computer operating system for graphic applications on general purpose computer 
systems today is the Windows® Operating System. Windows® assigns meaning to the cursor 
10 location. When the operator moves the cursor on top of a menu item and clicks, Windows® 

interprets the action as manifesting an intent to choose that menu item. The operator's path to that 
menu item, whether direct or circuitous, is irrelevant. Operators who can move toward a target 
accurately but cannot maintain the location indicated by a pointer on the target cannot effectively 
use standard Windows® applications through the conventional interface to these applications. 

15 

Often NMD operators cannot steady a pointer while operating a switch; the act of operating the 
switch triggers involuntary muscle activity pulling the cursor off target. For these operators, 
conventional selection by click is not practicable. Conventional selection by dwell also requires 
greater fine motor control than many NMD operators bring to this task. Operators with impaired 

20 ability to stop motion may overshoot their intended target. Operators whose voluntary muscle 
activity is accompanied by some involuntary muscle activity affecting their directional control 
cannot point accurately. Operators with involuntary tremors cannot maintain the location indicated 
by a pointer on a key image. Consequently, NMD operators who try to use on-screen keyboards 
often miss their target key images and accidentally select unintended key images. Following such 

25 an error, the operator must erase his erroneous selection by selecting the backspace or undo key. 
As the number of erroneous selections increases, the operator's productivity decreases markedly, 
since each error requires a correction in which there might be another error. 

Measures of an individual's fine motor control assist a physician or therapist in evaluating the 
30 effectiveness of a treatment program, including assistive technology, and in gauging the severity of 
a disability. Such measures help the physician or therapist in determining what treatment course to 
pursue and whether the severity of a certain disability justifies the risk of a particular treatment 
option, such as neurosurgery. 



13 



WO 96/30822 

One object of the invention is to facilitate selection of an option from a menu. 



PCT/US95/03591 



Another object of the invention is to indicate to an operator moving a cursor toward an option in a 
menu displayed by a computer system, which option the computer system believes the operator is 
5 moving toward. 

Another object of the invention is to display a menu on a display so that a large contiguous area on 
the display is not obstructed by the menu. 

10 Still another object of the invention is to make use, in computer access, of relatively unimpaired 
directional control in persons having impaired fine motor control. 

A further object of the invention is to indicate to an operator moving a cursor toward a dwell- 
selectable option in a menu, a changed selection threshold of the dwell-selectable option. 

15 

Yet another object of the invention is to more efficiently select an option from a menu on a display. 

Another object of the invention is to speed up selection of an option from a menu by an operator 
having impaired ability to operate a switch while the operator simultaneously keeps a cursor 
20 location within a region on a display. 

Another object of the invention is to help an operator with a disability control a pointer. 

Yet another object of the invention is to speed data entry by an individual with a disability. 

25 

A still further object of the invention is to facilitate artificial speech generation by a person having 
impaired speech due to a neurogenic muscular disorder. 

Yet another object of the invention is to facilitate device control by a person having a neurogenic 
30 muscular disorder. 

Another object of the invention is to measure an individual's ability to move one of the individual's 
body members in a direct path from a starting position to an ending position. 
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Yet another object of the invention is to facilitate computer access for an individual who cannot 
stop movement cleanly. 

E. Intersection 

5 

One object of the invention is to facilitate computer access by an operator having impaired ability 
to maintain a body member in a steady position. 

Another object of the invention is to facilitate the selection of a desired menu option by an operator 
10 having impaired fine motor control. 

A further object of the invention is to synthesize speech for an operator having impaired speech 
and impaired motor control. 

15 Yet another object of the invention is to use an operator's directional control in computer access. 

A still further object of the invention is, in selecting a menu option from a menu of dwell-selectable 
menu options, to compensate for an operator's impaired ability to maintain a body member in a 
steady position by using the operator's relatively intact motor capability . 

20 

F. Alignment 

Conventional on-screen keyboards do not compensate for NMD operators' inability to stop motion. 
Suppose the operator has been fitted with a head pointing device so that his head motion moves the 

25 cursor and that he's using the quaternary keyboard shown in Figures 3, 4 and 5. Assume further 
that, as he attempts to point to the quadrant containing the "j" key image (1308) in Figure 3, he is 
unable to stop on that quadrant and continues turning 20 more degrees to the left. There are two 
known ways of responding to this situation: (1) the cursor may continue to track the operator's 
motion and disappear from the display, leaving no indication to the operator of the location of the 

30 cursor and consequently causing some operator disorientation, or (2) the cursor may "stick", i.e. 
remain confined to the display, at, for example, point (1312). Conventional on-screen keyboards 
respond in this way. The operator's line of sight is now 20 degrees to the left of the cursor location. 
After the dwell period, the quadrant (1308) is selected and Figure 4 is displayed. The cursor hasn't 
moved. It is now at point ( 1 320). Assume that again the operator attempts to point to the quadrant 
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containing the "j" key image ( 1 324) in Figure 4. As the operator turns his head to the right, the 
cursor immediately moves with him. Thus, the operator's line of sight remains 20 degrees to the 
left of the cursor location as the cursor moves to the right across the display. The operator must 
watch the cursor out of his right eye. The problem is aggravated if either the operator cannot 

5 cleanly stop or if he drifts as he dwells. Assume that while attempting to dwell on quadrant (1324) 
the operator drifts 25 degrees past the bottom of the screen. His line of sight is now 25 degrees 
below and 20 degrees to the left of the cursor. To correct this misalignment in the conventional 
quaternary keyboard, the operator must turn his head to the right, "stick" the cursor against the 
right edge of the display, and continue turning 20 degrees until he has the cursor in his line of sight. 

10 Then he must lift his head until he sticks the cursor against the top edge and continue lifting 25 

degrees more. Alternatively, in this scenario, the operator could stick the cursor in the upper right 
corner of the display and simultaneously rotate his head up and to the right until he brought the 
cursor into his line of sight. 

15 Alignment is also a problematic for NMD operators who use a pointer, such as a mouse, with 
which the operator indicates by a location on a surface, e.g. a desk top, which corresponds to a 
desired location on the display, and achieve alignment by removing the pointer from the surface, 
e.g. lifting the mouse, moving the mouse, then replacing it on the surface. Due to impaired fine 
motor control, many NMD operators cannot remove a pointer from the surface and replace it on the 

20 surface at a desired location without unintentional movement or extraordinary effort. For these 
operators, alignment cannot be effectively achieved through conventional means. 

In summary, misalignment interferes with accurate pointing and the process of correcting for 
misalignment may result in the selection of unintended key images. 

25 

One object of the invention is to allow an operator to align a pointer with a location on a surface. 

Another object of the invention is to indicate to an operator a location on a surface with which he 
may align a pointer. 

30 

Still another object of the invention is to indicate to an operator when he may align a pointer with a 
location on a surface. 

A further object of the invention is to allow an operator having impaired motor control to align a 



16 



WO 96/30822 



PCT/US95/03591 



pointer with a cursor. 
G. Length Order 

5 As noted previously, one of the elements determining the menu option selection time is the time 
the operator requires to comprehend the menu options displayed. This time may be reduced if the 
operator can limit the number of menu options he searches in looking for his desired menu option. 
Conventional word prediction systems attempt to reduce this operator search time. The operator of 
a conventional word prediction system may, for example, select the letter "p". The system displays 

10 some number, say six, of the most frequently used words beginning with the letter "p". 

Conventionally these seven words are displayed either in alphabetic order or in order of frequency 
of use. Assuming the operator does not see his desired word on the display, he selects another 
letter, say "r". The system then displays the six most frequently used words beginning with the 
letters "pr". 

15 

Searching a displayed list of words in alphabetic order requires that the operator focus his attention 
on the selection task, as opposed to the information content of the conversation or other task the 
operator is engaged in. Further, determining whether a given word is alphabetically greater or 
lesser than a desired word takes substantial time, slowing the selection process. An alphabetically 
20 ordered list is of limited use to an individual who has below normal spelling ability, a frequent 

problem among individuals with impaired speech. Ordering words by frequency of use often does 
not limit the number of words the operator must search. The word at the bottom of the displayed 
list, for example, the sixth most frequently used word beginning with the letters "pr" may be a very 
common word, even though it is less frequently used than the other five displayed words. 

25 

One object of the Length Order aspect of the invention is to reduce the time an operator requires to 
comprehend displayed menu options. 

Another object of the Length Order aspect of the invention is to reduce operator search time. 

30 

Still another object of the Length Order aspect of the invention is to limit the number of menu 
options an operator searches for in looking for his desired menu option. 

A further object of the Length Order aspect of the invention is to speed data entry. 
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Yet another object of the Length Order aspect of the invention is to increase productivity in speech 
synthesis for an operator having impaired speech. 

H. Location Indication 

5 

The difficulties experienced by NMD operators in pointing to relatively small selectable regions 
have already been described. One approach to these difficulties is to enlarge the on-screen 
selectable region, illustrated by the quaternary expansion on-screen keyboard already described. 
Another approach is the conventional eye gaze system for a speech impaired individual, depicted in 

10 Figure 8. The system consists of a plexiglass frame (6352) having a centrally located aperture 
(6354). The eye gaze system is positioned between the speech impaired individual and person 
with whom the speech impaired individual is communicating. There are eight groups of five 
squares each on the plexiglass frame. Each square within each group of five squares is color 
coded, e.g. red, blue, green, yellow and clear, matching the color on each of the four corners of the 

15 plexiglass frame. The clear square matches the aperture (6354). All squares are labeled with 
symbols representing items to be communicated. These labels are not shown in Figure 8. The 
person with whom the speech impaired individual is communicating observes the eyes of the 
speech impaired individual to determine the target of the speech impaired individual's eye gaze. To 
communicate an item, the speech impaired individual gazes first toward the one of the eight groups 

20 of five squares, indicating that he wants to communicate one of the symbols in that group, and 

gazes second toward one of the four corners and aperture (6354) matching the color of the square 
labeled with the item to be communicated in the previously indicated group. 

Two types of selectable regions are conventionally used in a point and click menu interface in a 
25 graphical user interface. The first, shown in Figure 9, depicts a menu having three menu options, 
labeled "High", "Medium" and "Low", each displayed on a display (4807), each associated 
respectively with selectable regions (4801 ), (4803) and (4805), and each located adjacent the 
associated selectable region. Figure 10 depicts a menu having the same three menu options, each 
displayed on a display (4807), each associated respectively with selectable regions (4901 ), (4903) 
30 and (4905), and each intersecting the associated selectable region. In both these conventional 

menus, a menu option is selected by pointing to and clicking on the associated selectable region. 

Conventional menu hierarchies in automated systems, built from menus of the type shown in 
Figure 9 or Figure 1 0, require that the operator proceed sequentially through the steps of searching 
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menu options, selecting one of them, and, assuming a menu option including a submenu was 
selected, searching the submenu options, and selecting one of them. Where selection from menu 
hierarchies constitutes a substantial component of the operator's activities, the slowness of the 
selection process diminishes productivity. 

5 

Locating selectable regions or parts thereof outside the display, in accordance with the Perimeter 
Menu aspect of the invention, allows the large areas outside the display to be used, a major 
advantage for operators having impaired fine motor control who are unable to maintain a pointer on 
a small selectable region while selecting by click or by dwell. However, if menu options are 
10 displayed on the display near the perimeter of display and near their associated selectable region, 
the operator has an indication of the location of each selectable region but may not be able to see 
all the displayed menu options in a glance. Because an operator usually searches a displayed menu 
for his intended menu option, placing the menu only near the perimeter of the display may increase 
menu search time, thus increasing menu option selection time. 

15 

One object of the invention is to indicate to an operator of a menu system having selectable regions 
outside the display, the menu option associated with each selectable region and the location of each 
selectable region. 

20 Another object of the invention is to facilitate selection from a menu by an operator having 
impaired motor control. 

Still another object of the invention is to speed selection of a menu option from a menu and of a 
submenu option from a menu hierarchy. 

25 

A further object of the invention is to speed speech synthesis for a person having impaired speech 
and impaired motor control. 

1. Sound Match 

30 

Conventional speech recognition systems facilitate computer access for individuals unable to use a 
standard keyboard whose speech is relatively unimpaired, for example, an individual with 
quadriplegia, and hands-free computer access for able-bodied individuals. The operator of such a 
speech recognition system reads a menu option out loud, for example, "open file", and the system. 
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which includes sound receiving means, for example, a microphone coupled to a sound board 
having a Digital Signal Processor ("DSP"), receives the sound of the read menu option, digitizes 
the sound of the read menu option, and then provides the digitized sound to another component of 
the speech recognition system, sound matching means which includes an application program for 
5 matching the digitized sound to one of a plurality of sounds, each representing respectively the 

sound of a spoken menu option. The system determines which sound best matches the sound of the 
read menu option and selects the menu option associated with this best matched sound. 

Individuals whose speech is impaired are often unable to effectively use conventional speech 
10 recognition systems because they often cannot produce a large distinct variety of sounds 

characteristic of phonetic languages. For example, such an individual may produce similar sounds 
for the two consonants "t" and "d" so that these sound are indistinguishable to a conventional 
speech recognition system, or such an individual may not be able to consistently produce sounds 
distinguishable by a speech recognition system, resulting in false matches. Other symptoms of 
15 impaired speech, for example, similarities among certain phonemes and impaired ability to start or 
stop sound production appropriately, may substantially limit the variety of sounds distinguishable 
to a conventional speech recognition system an individual may consistently produce. 

Conventional speech recognition systems provide limited capabilities in languages rich in 
20 homophones, for example, Chinese, because in such languages, a distinct sound is often 

insufficient to specify a word, as is described in the Background Art section of the Ideographic 
Languages aspect of the invention. The problem may be briefly illustrated by an example. 
Suppose a Chinese data entry operator using a conventional speech recognition system speaks the 
phonetic unit "fu" with a particular intonation. This distinct sound may well have over 15 
25 homophones. Although the operator could use the keyboard to select one of these 1 5 homophones, 
this defeats the purpose of speech recognition, which is to facilitate hands-free computer access. 

One object of the invention is to facilitate selection from a menu, and, in particular, from a menu of 
homophones. 

30 

Another object of the invention is to facilitate speech synthesis and voice activated computer 
access by individuals with speech impairments. 

Still another object of the invention is to speed data entry in Ideographic Languages. 
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J. Ideographic Language 

The use of ideographs as the graphic symbols in written languages is found in many parts of the 
world. An ideograph, as used herein, is a graphic symbol used to represent an object, an idea or a 
5 word, without expressing, as in a phonetic system, the specific sounds forming the verbal 

expression of the object, idea or word. Ideographic languages include Chinese, Japanese and 
Korean. A graphic symbol, as used herein, includes, but is not limited to. each of the following: a 
letter of an alphabet, a Japanese kana, and an ideograph. For purposes of illustrating the concepts 
of the present invention specific reference will be made herein to a preferred embodiment of the 
10 system and method as it applies to the Chinese language. 

In modern Chinese, a repertoire of between 2500 and 3000 ideographs is necessary to achieve 
normal business adequacy in reading and writing, while the language itself has approximately 
50,000 ideographs that have been identified historically, with about 10.000 ideographs in current 

15 use. The conventional keyboard, with approximately 100 keys, is designed for languages with 
phonetic scripts, such languages having a small set of graphic symbols, i.e. letters. If such a 
keyboard were to be used in a corresponding manner for the direct input of Chinese ideographs, it 
would require many thousands of keys since, unlike western phonetic languages, Chinese has many 
thousands of ideographs. Selection of an ideograph from such a keyboard would require the 

20 operator to search a great many keys for the desired key, and thus be impracticably slow. 

Prior art methods for selecting Chinese ideographs make use of various ideograph classification 
systems known to Chinese speakers. The operator first specifies a class of ideograph, based on a 
first characteristic common to many ideographs. Ideographs having that common characteristic are 
25 displayed and the operator selects from among them, either directly, by selecting an individual 

ideograph, or indirectly, by specifying a second common characteristic usually dependent upon the 
first characteristic, thus further limiting the displayed ideographs to those having both the first and 
second common characteristics. In some prior art methods, the operator may continue to specify 
characteristics until he has specified a unique ideograph. 

30 

One ideograph classification system is called the Pin Yin System. This classification system uses 
the phonetic structure of the Chinese language. In spoken Chinese there are approximately 412 
basic phonetic units, each having a monosyllabic sound, for example, "nee", "how" and "ma". Four 
intonations can potentially be applied to each phonetic unit, resulting in approximately 1280 
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distinct sounds. With 10,000 ideographs in current use, each represented by one of approximately 
1280 distinct sounds, it is evident that many Chinese ideographs are homophones, i.e. have the 
same sound. Over 80% of Chinese ideographs have homophones. The Pin Yin System uses this 
limited number of phonetic units as the basis for its classification. Ideographs which are 
5 homophones are classified together; the common characteristic of the Pin Yin System is the distinct 
sound. 



According to the Pin Yin and Zhu Yin coding methods, known in the prior art. the operator 
specifies a distinct sound using a keyboard labeled with symbols representing the Latin alphabet 

10 (Pin Yin method) or Chinese phonetic units (Zhu Yin method). The first key operation or sequence 
of key operations specifies the phonetic unit. The second key operation specifies the intonation. In 
general, less than 15 ideographs have this sound, though in some cases there are many more 
homophones. These are displayed and the user selects from among them. In such cases, the 
operator, depending upon the system, may page through matching ideographs or specify another 

15 common characteristic to further limit the number of ideographs displayed. A common 

characteristic which may be used at this stage exploits another feature of the Chinese language. 
The majority of Chinese words are expressed by a combination of two ideographs, the meaning of 
the paired ideographs has its own meaning which may or may not be related to that of the 
constituent ideographs. Assuming the operator has specified a first distinct sound matching 40 

20 ideographs, he may specify a second distinct sound which alone may match, for example, 20 

ideographs, but there may be only two ideograph pairs having the specified first and second distinct 
sounds in that order. Thus, a second common characteristic may limit matching ideograph pairs to 
a number sufficiently small for the operator to efficiently search and select from, or may uniquely 
specify an ideograph pair. Another common characteristic the operator may specify to limit the 

25 number of matching ideographs is a meaning or meaning class to which one or more sequences of 
one or more ideographs belong. 

Yet another feature of the Chinese language which may be exploited to limit the number of 
matching sequences of ideographs is the ideograph block. An ideograph block is a sequence of 
30 four ideographs which together has its own meaning which may or may not be related to that of the 
constituent ideographs. As above, where the operator specified a distinct sound for the second of 
two ideographs of an ideograph pair, so may the operator specify a distinct sound for the second, 
third and/or fourth ideograph of an ideograph block, to limit the number of matching ideograph 
blocks. 
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Another conventional ideograph classification system makes use of a classification of parts of 
ideographs. Ideographs are built from a set of 214 components, called radicals. Different radicals, 
perhaps placed within different locations within an ideograph, are combined to create an ideograph. 
According to the Chan Jie coding method, known in the prior art, the operator specifies one or 
5 more radicals appearing in the ideograph he wishes to enter. He may, for example, use a keyboard 
having at least 214 keys, each corresponding to a radical, or may actuate a sequence of keys, the 
sequence corresponding to a radical. Other common characteristics the operator may specify to 
limit the number of matching ideographs include a phonetic unit, the first brush stroke, and the last 
brush stroke used to draw the ideograph. 

10 

Another conventional ideograph classification system makes use of a classification of parts of 
ideographs. According to the Four Corner coding method, known in the prior art, the operator 
specifies the classification of the four corners of the ideograph he wishes to enter. Other common 
characteristics the operator may specify to further limit the number of matching ideographs include 
1 5 the number of horizontal strokes used to draw the ideograph, and the classification of a certain part 
of the ideograph above the lower right corner. 

Yet another conventional ideograph classification system makes use of a classification ideographs 
based on the basic strokes from which each ideograph is built. In Chinese, there are a limited 
20 number of basic strokes, each ideograph being composed of between 1 and 33 such strokes. 

Ideographs may be classified by a small number of basic strokes, preferably according to strict 
rules regarding the order of stroke entry. In one conventional application of this coding method, 
the operator specifies only the first and last basic strokes of the desired ideograph, then selects 
from a display of all ideographs sharing this first-last basic stroke combination. 

25 

Japanese is somewhat more complicated than Chinese. In addition to ideographs, the Japanese 
language uses graphic symbols called kana, which includes hiragana and katakana. In written 
Japanese, ideographs are frequently combined with kana. Kana may be may specified phonetically, 
for example, to designate the hiragana pronounced "ko" an operator of a Japanese word processing 
30 system may type "k" and then "o" on a Latin alphabetic keyboard or may type a single key 

associated with this hiragana. Kana has multiple uses in a Japanese word processing system. 
Kana may represent itself, since kana may stand alone in Japanese text. Alternatively, kana may 
be used to specify Japanese ideographs, either by specifying the radicals which compose Japanese 
ideographs or by specifying the pronunciation of Japanese ideographs. A sequence of phonetic 
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units specified by kana may represent that sequence of kana, a single Japanese ideograph, multiple 
Japanese ideographs, or a combination of one or more Japanese ideographs and one or more kana. 
In addition, a single Japanese ideograph may have multiple pronunciations, including a Japanese 
pronunciation and a Chinese pronunciation, and may have multiple kana spellings. 

5 

Conventional word processing systems for ideographic languages suffer from certain deficiencies. 
First, in systems where the operator specifies common characteristics until he has uniquely 
specified an ideograph, the operator must be extensively trained in the particular classification 
system. Depending upon the system, the operator may need to know, for example, how may 

10 horizontal brush strokes are required to draw a desired ideograph, or each of the 214 radicals and 
the encoding of each of them on a keyboard having less than 2 1 4 keys. Second, in systems where 
the operator uses both hands on the keyboard to specify a common characteristic, then selects from 
among ideographs, ideograph pairs or ideograph blocks by operating a function key or by pointing 
to one of the options with a mouse or other hand operated pointer and then operating a switch, the 

15 operator lifts one of his hands from the keyboard, makes the selection and then moves his hand 
back to the keyboard to specify another common characteristic. This sequence occurs often and 
contributes to the slow average rate of word entry (approximately 20 words per minute) for 
Chinese relative to alphabetic languages. Another problem in these systems is that the display of 
ideographs for selection may obscure part of the image of the previously entered ideographs or 

20 other information on the display. 

Another drawback of many word processing systems for ideographic languages relates to the ease 
of copying a document. Ideally, the operator concentrates on the document to be copied, only 
occasionally scanning text he has input. For those word processing systems that display ideographs 
25 on a display for the operator's selection, the operator must frequently shift his gaze from the 

document to the display and back again. The operator cannot concentrate on both the document 
and the display simultaneously. 

Ideographs, as used herein, also include the symbols of symbol sets used for communication by 
30 individuals who have hearing, speech or language impairments, for teaching literacy skills to those 
lacking them, including pre-literate children and individuals with intellectual disabilities, and for 
international written communication. These symbol sets include, but are not limited to, each of the 
following: Picture Communication Symbols, Rebus, Picsym. Pictogram Ideogram Communication 
Symbols, Yerkish. Blissymbolics and depictions of the signs of a manual sign language. Examples 
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of symbols of the Picture Communication Symbols. Rebus, Picsyms, and Blissymbolics symbol 
sets are shown in Figure 1 1, Pictogram Ideogram Communication Symbols in Figures 12(a) - 
12(d) and Yerkish in Figures 13(a) - 13(j). Picture Communication Symbols, Rebus, Picsyms, 
Pictogram Ideogram Communication Symbols, Yerkish, and Blissymbolics are each described in 
5 Beukelman, David R. & Mirenda, Pat, Augmentative and. Alternative Communication. 

Management Of Severe Communication Disorders in Children and Adults . Paul H. Brookes 
Publishing Co.. 1992, pp. 22-29. 

Individuals who have not acquired or who have lost their literacy skills may use symbolic symbol 
10 sets in learning to read. If the individual lacks fine motor control, for example, due to cerebral 
palsy, the individual's disability may inhibit the acquisition of literacy skills by, for example, 
inhibiting repetition of an exercise by the individual, by limiting the individual's ability to 
participate in the classroom, or by making skill assessment by a teacher difficult so that the teacher 
may incorrectly believe that remediation is necessary or that a particular skill has been mastered. If 
15 the individual also has impaired speech, literacy acquisition is more difficult still. 

Conventional literacy training systems for individuals who are unable to use a standard keyboard or 
mouse use switch access, often in combination with scanning. As already described, scanning is an 
extremely slow process. Moreover, as the number of symbols in the symbol set increases, the time 

20 required to select a symbol also increases. Of the symbol sets mentioned above, Picture 

Communication Symbols contains approximately 1800 symbols, Rebus contains approximately 
800 symbols, Picsyms contains approximately 1800 symbols, Pictogram Ideogram Communication 
Symbols contains approximately 400 symbols and Blissymbolics contains approximately 1400 
symbols. When using a system with a static display, the operator may expend considerable time 

25 and effort finding the desired symbol; when using a system with a dynamic display, the operator 
may expend considerable time effort memorizing and recalling where a particular symbol is 
located within a hierarchy of symbols. This time and effort generally does not contribute to the 
acquisition of literacy skills. 

30 One object of the invention is to display a menu of sequences of one or more ideographs on a 
display so that a large contiguous area on the display is not obstructed by the menu. 

Another object of the invention is to facilitate ideograph entry in word processing systems for the 
Chinese, Japanese and Korean languages. 

25 
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Still another object of the invention is to speed selection of sequences of graphics including one or 
more ideographs. 



Yet another object of the invention is to allow an operator of a word processing system for an 
5 ideographic language to select a sequence of one or more ideographs without lifting either hand 
from the keyboard. 

A further object of the invention is to indicate to an operator the progress toward selection of a 
dwell-selectable sequence of one or more ideographic characters. 

10 

A still further object of the invention is to synthesize speech for an operator having impaired 
speech. 

15 Additional objects, advantages and novel features of the invention will be set forth in part in the 
description which follows, and in part will become apparent to those skilled in the art upon 
examination of the following or may be learned by practice of the invention. The objects and 
advantages of the invention may be realized and attained by means of the instrumentalities and 
combinations particularly pointed out in the appended claims. 

20 

Disclosure of Invention 
A&B Perimeter Menu and Confinement 

25 

According to the present invention, the objects mentioned in the Background Art section describing 
the above aspects of the invention and other objects and advantages are attained by an apparatus for 
selecting a menu option from a menu. Each menu option is associated respectively with a 
selectable region on a display area. The apparatus includes (a) the display area; (b) means for 
30 delimiting a plurality of selectable regions, each of the selectable regions associated respectively 
with a menu option and each of the selectable regions. including an invisible subregion adjacent the 
display area and a visible subregion on the display area, the plurality of visible subregions together 
at least partially circumscribing a region on the display area; (c) movement related signal receiving 
means for receiving a movement related signal indicating a location; and (d) selection means for 
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selecting, in response to a selection event, the menu option associated with the selectable region 
intersected by the location indicated by the movement related signal. 



Additionally, the objects mentioned in the Background Art section describing the above aspects of 
5 the invention and other objects and advantages are attained by a method of selecting a menu option 
from a menu. The menu options are each respectively associated with a selectable region on a 
surface which includes a display area. The selectable regions together at least partially 
circumscribe a region on the display area. An operator indicates a location on the surface with a 
body member. The method comprises the steps of: (a) confining the location indicated by the 
10 body member of the operator to the display area; and (b) selecting, in response to the period of one 
or more intersections of the location indicated by the movement related signal and any one 
selectable region equalling or exceeding a predetermined period, the menu option associated with 
the intersected selectable region. 

1 5 The method and apparatus of the aspect of the invention disclosed above and other aspects of the 
invention permits an NMD operator to utilize for computer access his relatively unimpaired motor 
capabilities, in particular, gross motor control, in lieu of, or as a supplement to. his impaired motor 
capabilities. The method and apparatus of the invention also reduces the interference of menu 
option displays with the display of information. 

20 

C. Dwell 

According to the present invention, the objects mentioned in the Background Art section describing 
the above aspect of the invention and other objects and advantages are attained by an apparatus for 
25 indicating dwell time comprising a surface having a region thereon; movement receiving means for 
receiving a movement related signal indicating successive locations; and indicating means for 
indicating at least the duration of a period of intersection of two or more of the successive locations 
indicated by the movement related signal and the region. 

30 Additionally, the objects mentioned in the Background Art section describing the above aspect of 
the invention and other objects and advantages are attained by a method of indicating dwell time 
comprising the steps of displaying a selectable region on a surface; receiving a movement related 
signal indicating a first location intersecting the region and. at a later time, a second location 
intersecting the region; and indicating the difference between the time of the second location and 
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the time of the first location. 
D. Path Directness 

5 According to the present invention, the objects mentioned in the Background Art section describing 
the above aspect of the invention and other objects and advantages are attained by an apparatus for 
selecting a menu option from a menu. Each menu option is associated respectively with a 
selectable region shown on a surface. The apparatus includes (a) means for moving a cursor on the 
surface in response to a movement related signal, for example, a computer program which moves a 

10 cursor on the surface reflecting the operator's movement of a mouse: and (b) means for selecting 
the menu option associated with the selectable region most nearly along a cursor path ( 1 ) prior to 
an intersection of the cursor and the selectable region most nearly along the cursor path, (2) upon 
intersection of the cursor and the selectable region most nearly along the cursor path, or (3) in 
response to the period of intersection equalling or exceeding a predetermined period. 

15 

Also according to the present invention, the objects mentioned in the Background Art section 
describing the above aspect of the invention and other objects and advantages are also attained by a 
method for selecting a menu option from a menu. Each menu option is associated respectively with 
a selectable region shown on a surface. The method includes the steps of (a) displaying the 

20 selectable regions on the surface, (b) detecting a movement related signal and in response moving a 
cursor on the surface, and (c) selecting the menu option associated with the selectable region most 
nearly along a cursor path (1) prior to an intersection of the cursor and the selectable region most 
nearly along the cursor path, (2) upon intersection of the cursor and the selectable region most 
nearly along the cursor path, or (3) in response to the period of intersection equalling or exceeding 

25 a predetermined period. 

In accordance with the present invention, the objects mentioned in the Background Art section 
describing the above aspect of the invention and other objects and advantages are attained by an 
apparatus for measuring an individual's ability to move one of the individual's body members 
30 directly from a starting position to an ending position. The apparatus includes: (a) receiving means 
for receiving a sampling among data indicative of successive positions of the body member; and 
(b) measurement means for measuring any deviation of the path indicated by the received position 
data from a direct path between the starting position and the ending position. 
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The objects mentioned in the Background Art section describing the above aspect of the invention 
and other objects and advantages are also attained, according to the present invention, by a method 
for measuring an individual's ability to move one of the individual's body members directly from a 
starting position to an ending position. The method includes the steps of: (a) receiving a sampling 
5 among data indicative of successive positions of the body member; and (b) measuring any 

deviation of the path indicated by the received position data from a direct path between the starting 
position and the ending position. 

E. Intersection 

10 

According to the present invention, the objects mentioned in the Background Art section describing 
the above aspect of the invention and other objects and advantages are attained by an apparatus for 
selecting an option from a menu. The apparatus comprises cursor movement means for receiving a 
movement related signal and for moving a cursor on a display responsive to the received movement 

15 signal; delimit means for delimiting with respect to the display a first plurality of regions and a 
second plurality of selectable regions, each of the first plurality of regions associated respectively 
with one of the second plurality of selectable regions, each of the second plurality of selectable 
regions associated respectively with a menu option, wherein at least one of the first plurality of 
regions is not coterminous with its associated one of the second plurality of selectable regions; and 

20 selection means, responsive to an intersection of the cursor and any one of the first plurality of 

regions and to a selection event associated with the one of the second plurality of selectable region 
associated with the intersected one of the first plurality of regions, for selecting the menu option 
associated with the selectable region associated with the selection event. 

25 According to the present invention, the objects mentioned in the Background Art section describing 
the above aspect of the invention and other objects and advantages are also attained by a method of 
selecting an option from a menu, said method comprising the steps of: receiving a movement 
related signal and moving a cursor on a surface responsive thereto; delimiting a first plurality of 
regions and a second plurality of selectable regions with respect to a surface, each of the first 

30 plurality of regions associated respectively with one of the second plurality of selectable regions, 
each of the second plurality of selectable regions associated respectively with a menu option; 
wherein at least one of the first plurality of regions is not coterminous with its associated one of the 
second plurality of selectable regions; and responsive to an intersection of the cursor and any one 
of the first plurality of regions and to a selection event associated with the one of the second 
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plurality of selectable region associated with the intersected one of the first plurality of regions, 
selecting the menu option associated with the selectable region associated with the selection event. 



F. Alignment 

5 

Confining the cursor to the display or to a polygon on the display, in accord with the Perimeter 
Menu aspect of the invention, facilitates selection from a perimeter menu. However, for an 
operator who overshoots a perimeter menu option, confining the cursor to the display results in a 
loss of alignment between the cursor and the location indicated by the operator, necessitating an 
10 apparatus and method for realignment. 

According to the present invention, the objects mentioned in the Background Art section describing 
the above aspect of the invention and other objects and advantages are attained by an apparatus, 
including a surface on which may be displayed a moveable cursor, said apparatus allowing an 
15 operator to align a pointer with a predetermined location on the surface. The apparatus includes a 
display means for displaying the cursor on the surface; movement related signal receiving means 
for receiving a movement related signal; and control means for moving the cursor in response to 
the movement related signal and for thereafter inhibiting movement of the cursor for a first period 
of time in response to an operator action detected by the movement related signal receiving means. 

20 

According to the present invention, the objects mentioned in the Background Art section describing 
the above aspect of the invention and other objects and advantages are also attained by a method, 
for use in a system comprising a pointer and a surface on which is displayed a first cursor whose 
movement on the surface is responsive to the movement of the pointer, of aligning the first cursor 
25 with a predetermined location on the surface. The method includes the steps of: (a) producing the 
first cursor on the surface; (b) moving the first cursor responsive to the movement of the pointer, 
said movement restricted to a region on the surface; and then (c) responsive to an operator action, 
producing a second cursor at a predetermined location on the surface for a period of time; and (d) 
moving the first cursor in or in close proximity to the predetermined location.. 

30 

G. Length Order 

According to the present invention, the objects mentioned in the Background Art section describing 
the above aspect of the invention and other objects and advantages are attained by an apparatus for 



30 



WO 96/30822 



PCT/US95/03591 



use in a menu interface system. The apparatus orders a plurality of named menu options at least in 
part according to the order of the length of each of the names of the name menu options, and then 
displays the named menu options in that order 

5 According to the present invention, the objects mentioned in the Background Art section describing 
the above aspect of the invention and other objects and advantages are also attained by a method of 
data entry for use with a computer. According to the method, a plurality of named menu options 
are displayed at least in part according to the order of the length of each of the names of the name 
menu options. Each named menu option is associated respectively with a sequence of one or more 
10 characters. A selection event is associated with one of the named menu options and the sequence 
of one or more characters associated with the selected named menu option is input to an application 
program executing on the computer. 

H. Location Indication 

15 

According to the present invention, the objects mentioned in the Background Art section describing 
the above aspect of the invention and other objects and advantages are attained by an apparatus for 
indicating the location of the selectable region associated with a menu option. Each menu option is 
associated respectively with a selectable region. The apparatus includes (a) means for at least 
20 partially delimiting the selectable regions; and (b) indication means for displaying each menu 
option such that the displayed menu option indicates the location of the associated selectable 
region. None of the displayed menu options intersects the selectable region associated therewith. 

According to the present invention, some of the objects mentioned in the Background Art section 
25 describing the above aspect of the invention and other objects and advantages are also attained by a 
method of indicating the location of the selectable region associated with a submenu option in a 
menu hierarchy. The method comprises the steps of: at least partially delimiting the plurality of 
selectable regions; and displaying a plurality of submenu indicating regions. Each submenu 
indicating region is associated respectively with one of the submenu options and each submenu 
30 indicating region is displayed in a manner indicating the location of the selectable region associated 
with the submenu option associated with the submenu indicating region. 

I. Sound Match 
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According to the present invention, the objects mentioned in the Background Art section describing 
the above aspect of the invention and other objects and advantages are attained by an apparatus for 
selecting a menu option from a plurality of menu options. Each menu option is displayed on a 
display and associated respectively with a sound. The apparatus includes: (a) means for displaying 
5 the menu on the display: (b) means for receiving a sound signal; (c) means for matching the 

received sound signal to any one of sounds: and (d) means for selecting the menu option associated 
with the matched sound. 

According to the present invention, some of the objects mentioned in the Background Art section 
10 describing the above aspect of the invention and other objects and advantages are also attained by a 
method of selecting a sequence of one or more graphic symbols from a plurality of sequences of 
one or more graphic symbols, one or more sequences of the plurality of sequences including one or 
more ideographs. The method comprises the steps of displaying on a display the plurality of 
sequences, each of the plurality of sequences having a common characteristic and each associated 
15 respectively with a sound indicator: matching a sound to any one of the plurality of sound 

indicators: and selecting the sequence of the plurality of sequences associated with the matched 
sound indicator. 

J. Ideographic Languages 

20 

According to the present invention, the objects mentioned in the Background Art section describing 
the above aspect of the invention and other objects and advantages are attained by a method for 
selecting a single sequence of one or more ideographs from a menu of several such sequences. 
Each sequence is respectively associated with a selectable region on a surface which includes a 
25 display area The selectable regions together at least partially circumscribe a region on the display 
area. An operator indicates a location on the surface with a body member. The method comprises 
the steps of: (a) displaying the selectable regions on the surface so that they at least partially 
circumscribe a region on the display area; and (b) selecting, in response a selection event, the 
sequence indicated by the operator. 

30 

Still other objects and advantages of the present invention will become readily apparent to those 
skilled in this art from the following detailed descriptions, wherein 1 have shown and described the 
preferred embodiment of each aspect of the invention, simply by way of illustration of the best 
mode contemplated by me of carrying out each aspect of my invention. As will be realized, each 
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aspect of the invention is capable of other and different embodiments, and its several details are 
capable of modifications in various obvious respects, all without departing from the invention. 
Accordingly, the drawings and descriptions are to be regarded as illustrative in nature, and not as 
restrictive. 

5 

Brief Description of Drawings 
FIG. 1 is an illustration of a display showing a conventional on-screen keyboard. 

10 

FIG. 2 is an illustration of a display showing a conventional on-screen keyboard and output from a 
word processing application program. 

FIG. 3, 4 and 5 are each illustrations of the display of each step of letter selection using a 
15 conventional quaternary on-screen keyboard. 

FIG. 6 is an illustration of a display showing a conventional quaternary on-screen keyboard and 
output from two application programs. 

20 FIG. 7 is an illustration of a conventional device implementing joystick patterns. 

FIG. 8 is an illustration of a display of a conventional eye gaze system. 

FIG. 9 and 10 are each illustrations of a display showing a conventional menu. 

25 

FIG. 1 1 depicts examples of symbols from the Picture Communication Symbols, Rebus, PicSym 
and Blissymbols symbol sets. 

FIG. 12(a) - FIG. 12(d) depict examples of symbols from the Pictogram Ideogram Communication 
30 Symbols symbol set. 

FIG. 13(a) - FIG. 13(j) depict examples of symbols from the Yerkish symbol set. 

FIG. 14 is a block diagram of a computer which may be utilized in accordance with the present 
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invention. 

FIG. 1 5 is a block diagram of a speech synthesis system which may be utilized in accordance with 
the present invention. 

5 

FIG. 16 is an illustration of software components of an apparatus in accordance with an 
embodiment of the Perimeter Menu aspect of the invention. 

FIG. 17 and 18 are each illustrations of a display and structures in accordance w ith an embodiment 
10 of the Perimeter Menu aspect of the invention. 

FIG. 19 is an illustration of a display and structures in accordance with another embodiment of the 
Perimeter Menu aspect of the invention. 

15 FIG. 20 is an illustration of a display and structures in accordance with still another embodiment of 
the Perimeter Menu aspect of the invention. 

FIG. 21 is an illustration of a display and structures in accordance with another embodiment of the 
Perimeter Menu aspect of the invention. 

20 

FIG. 22 is an illustration of a display and structures in accordance with yet another embodiment of 
the Perimeter Menu aspect of the invention. 

FIG. 23 is an illustration of an apparatus in accordance with a further embodiment of the Perimeter 
25 Menu aspect of the invention. 

FIG. 24 and 25 are each illustrations of a display and structures in accordance with another 
embodiment of the Perimeter Menu aspect of the invention. 

30 FIG. 26 and 27 illustrate an apparatus in accordance with still another embodiment of the Perimeter 
Menu aspect of the invention. Figure 26 depicts a front view of the apparatus. Figure 27 depicts a 
cut away view from the top of the apparatus. 

FIG. 28 is a top view of a headrest in accordance with an embodiment of the Perimeter Menu 
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aspect invention. 

FIG. 29 and 30 illustrate the state table aPocketFsm in accordance with the preferred embodiment 
of the Perimeter Menu aspect of the invention. 

5 

FIG. 31 is an illustration of a display and structures in accordance with the preferred embodiment 
of the Confinement aspect of the invention. 

FIG. 32 is an illustration of a display and structures in accordance with another embodiment of the 
10 Confinement aspect of the invention. 

FIG. 33 is an illustration of a display and structures in accordance with another embodiment of the 

Confinement aspect of the invention. 

15 FIG. 34 is an illustration of an apparatus in accordance with the Dwell aspect of the invention and 
with the Path Directness aspect of the invention. 

FIG. 35 is an illustration of another apparatus in accordance with the Dwell aspect of the invention. 

20 FIG. 36 is an illustration of still another apparatus in accordance with the Dwell aspect of the 
invention. 

FIG. 37 is an illustration of yet another apparatus in accordance with the Dwell aspect of the 
invention. 

25 

FIG. 38 is an illustration of another apparatus in accordance with the Dwell aspect of the invention. 

FIG. 39 is an illustration of a display and structures in accordance with the preferred embodiment 
of the Path Directness aspect of the invention. 

30 

FIG. 40 is an illustration of a display and structures in accordance with an embodiment of the Path 
Directness aspect of the invention. 

FIG. 41 is an illustration of a display and structures in accordance with another embodiment of the 
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Path Directness aspect of the invention. 

FIG. 42 is an illustration of a display and structures in accordance with another embodiment of the 
Path Directness aspect of the invention. 

5 

FIG. 43 is an illustration of a display and structures in accordance with another embodiment of the 
Path Directness aspect of the invention. 

FIG. 44. 45 and 46 are each illustrations of a display and structures in accordance with the 
10 preferred embodiment of the Intersection aspect of the invention. 

FIG. 47 and 48 are each illustrations of a display and structures in accordance with another 
embodiment of the Intersection aspect of the invention. 

15 FIG. 49, 50 and 5 1 are each illustrations of a display and structures in accordance with the 
preferred embodiment of the Alignment aspect of the invention. 

FIG. 52 is an illustration of a display and structures in accordance with the preferred embodiment 
of the Location Indication and the Length Order aspects of the invention. 

20 

FIG. 53 is an illustration of a display and structures in accordance with an embodiment of the 
Location Indication aspect of the invention. 

FIG. 54 and 55 are each illustrations of a display and structures in accordance with another 
25 embodiment of the Location Indication aspect of the invention. 

FIG. 56 is an illustration of a display and structures in accordance with a still further embodiment 
of the Location Indication aspect of the invention. 

30 FIG. 57 is an illustration of a display and structures in accordance with a further embodiment of the 
Location Indication aspect of the invention. 

FIG. 58 and 59 are illustrations of a display and structures in accordance with the preferred 
embodiment of the Sound Match aspect of the invention. 
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FIG. 60 is an illustration of a display and structures in accordance with another embodiment of the 
Sound Match aspect of the invention. 

FIG. 61 is an illustration of a display and structures in accordance with another embodiment of the 
5 Sound Match aspect of the invention. 

FIG. 62 is a block diagram of a speech recognition system which may be utilized in accordance 
with the Sound Match aspect of the invention. 

10 FIG. 63 is an illustration of software components of an apparatus in accordance with an 
embodiment of the Sound Match aspect of the invention. 

FIG. 64 is an illustration of a display and structures in accordance with the preferred embodiment 
of the Ideographic Language aspect of the invention. 

15 

FIG. 65 is an illustration of a display and structures in accordance with another embodiment of the 
Ideographic Language aspect of the invention. 

FIG. 66 is an illustration of a display and structures in accordance with still another embodiment of 
20 the Ideographic Language aspect of the invention. 

FIG. 67 is an illustration of a display and structures in accordance with yet another embodiment of 
the Ideographic Language aspect of the invention. 

25 

Best Mode for Carrying Out the Invention 

The hardware and software operating environment of the preferred embodiment of all aspects of 
the invention will now be described with reference to a particular embodiment of the invention, 
30 hereinafter "prototype". The prototype of the invention illustrates the best mode of practicing each 
aspect of the invention known by me except where a preferred mode is described. 

Figures 15 depicts a block diagram of the hardware components of the prototype (2214), including 
a conventional general purpose computer system (2218), an optional pointer (2202), an optional 
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printer (2220) and a speech synthesizer (2206). The general purpose computer system (22 1 8) 
includes a conventional computer system (2116), a storage unit (2208), a keyboard (2210), and a 
diskette drive (2216). Figure 14 depicts a block diagram of the conventional computer system 
(2116). including a processing unit (2 1 02) and a display (2 1 1 2). The processing unit (2 1 02) 

5 includes a processor (2104), a memory (2106) and control circuitry (2108). The prototype 
employs the Toshiba T6400DXC general purpose computer system manufactured by Toshiba 
Corporation, Kawasaki, Japan. However, the T6400DXC is preferably substituted with the IBM 
ThinkPad 755C computer system, part number 9545F0C. manufactured by IBM Corporation, 
Armonk. New York. USA. because the former requires a 1 10VAC power source while the latter is 

10 powered by an integral battery. An integral battery allows an NMD operator to use the system 
when a 100VAC source is not available or when attaching to a 1 10VAC power source is 
inconvenient. The prototype further includes a head mounted pointer communicating via an 
infrared link with the computer system so that there are no cables tethering the operator to the 
computer system. Any cable between the operator and the computer system would have to be 

15 connected, probably by an attendant since the operator may lack the fine motor skills required to 
make such a connection. Preferably computer access can be accomplished independently by the 
operator. The choice of a pointing device is primarily dictated by the particular capabilities of the 
operator. Usually the best pointing device for a particular operator is the one drawing on that 
operator's best motor control. For example, if an operator's foot control is superior to his head 

20 control, a pointing device using his foot is preferably to a head pointer. 

The prototype employs the Remote Headmaster® manufactured by the Prentke Romich Company, 
Wooster. Ohio, USA. However, the combination of the HeadMaster® Plus, part number HM-1P, 
HeadMaster® Plus Remote Adapter, part number HM-RA. and HeadMaster® Plus Laptop 
25 Adapter, part number HM-LA, all available from the Prentke Romich Company, is preferable 

because the headset is more comfortable and the HeadMaster® Plus ultrasonic transmitter mounts 
more easily on a laptop computer system than the Remote Headmaster ultrasonic transmitter. 

The preferred embodiment further includes a battery powered printer, the MobileWriter®, part 
30 number 730879, manufactured by Mannessmann Tally Corporation, Kent, Washington, USA and a 
speech synthesizer, the Multivoice Speech Synthesizer, part number MV2-SS, manufactured by 
The Institute on Applied Technology, Children's Hospital, Boston, Massachusetts, USA. 

The pointer (2202) is a device which provides data concerning the relative or absolute position of 
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the operator or any body member of the operator. The display (2112) and pointer (2202) together 
provide for the interactive nature of the general purpose computer system (2218) in that, in accord 
with the various aspects of the invention, the interpretation that the processor (2104) gives to a 
certain pointer action made by the operator depends, in the majority of situations, upon what is 
5 being displayed to the operator at that time. 

The prototype (2214) shown in Figure 15 further includes a keyboard (2210), which functions to 
provide input from an able-bodied operator to the general purpose computer system (2218). The 
keyboard (2210) is useful for configuration, diagnostic and backup purposes, functions which are 

10 performed relatively infrequently and usually require an able-bodied person for ancillary activities, 
for example, loading backup media into the general purpose computer system. The prototype 
(2214) also optionally includes a printer (2220) which functions to provide hard copy output of 
data developed or stored in the general purpose computer system, and a speech synthesizer (2206), 
which functions to provide speech output for utterances and words composed using or retrieved 

15 from the general purpose computer system (2218). 

The couplings between the devices depicted in Figure 1 5 may be made by any means which 
permits the orderly and timely exchange of data across the interface. In the preferred embodiment, 
the interfaces between the pointer (2202) and the general purpose computer system (2218) and 
20 between the general purpose computer system (221 8) and the speech synthesizer (2206) conform to 
the Electronic Industries Association RS-232 interface specification. The interface between the 
general purpose computer system (221 8) and the printer (2220) conform to the Centronix 50 pin 
parallel interface specification. 

25 The software component of the prototype are stored in memory (2106) and executed on the 

processing unit (21 02). The software component of the prototype, depicted in Figure 16, include a 
software driver (1202), an operating system (1204), an optional database program (1210), and the 
prototype access program code and data, hereinafter collectively referred to as the "access 
program" (1206). In the preferred embodiment, one or more application programs (1208) may also 

30 execute on the processing unit (2102) and accept control and data from the access program (1206) 
via the operating system (1204). The software driver (1202) of the prototype is the Logitech 
Mouse Driver included with Windows® version 3.1. The operating system (1204) of the prototype 
is Windows® version 3.1 in combination with MS-DOS® version 6.2. Hereinafter, the operating 
system is referred to simply as "Windows®", available from Microsoft Corporation, Redmond. 
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Washington, USA. 

The optional database program (1210) is described in the detailed description of the Length Order 
aspect of the invention. The prototype access program (1206) is described in detail below. 

5 

As stated earlier, the software components of the prototype are stored in memory (2106). 
Depending on the capacity of memory (2106) and the size of the application programs, portions of 
these programs may be transferred as needed between memory (2106) and the storage unit (2206) 
or between memory (2106) and a diskette in the diskette drive (2216) depicted in Figure 15. The 
10 basic function of the storage unit (2206) and the diskette drive (22 1 6) is to store programs and data 
that are employed by the general purpose computer system (2218) and which may readily be 
transferred to the memory (2106) when needed. 

It is to be understood that components others than those used in the prototype may be utilized in 
15 accordance with the invention. It is only necessary that the substitute component or components 
have the capacity to carry out the functions described. For example, the processing unit of the 
general purpose computer system may be substituted with a microprocessor coupled to custom 
electronics for performing the functions of the various aspects of the invention, or the color display 
of the prototype may be substituted with a monochrome display. 

20 

A. Perimeter Menu 

The preferred embodiment of the Perimeter Menu aspect of the invention will now be described in 
detail from a functional perspective using an example. This description refers to selectable regions 

25 which include one or more subregions. A selectable region is a region, delimited with respect to a 
display or a surface, and associated with a menu option which may be selected, usually by a 
selection event. A subregion is a selectable region that is included within another selectable region. 
Thus a subregion is, by itself, a selectable region. Assuming that a certain selectable region 
includes subregions A and B, dwell time on subregions A and B may be combined, for example, by 

30 summing, so that dwelling on either subregion A or B or a combination of both for the selection 
threshold period selects the menu option associated with the selectable region. 

Reference will now be made to Figures 17 and 18 which depict an example of the preferred 
embodiment of the Perimeter Menu aspect of the invention. Figure 1 7 shows the display (21 12) of 
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a genera] purpose computer system (22 1 8 in Figure 15) and eight selectable regions. Each of the 
eight selectable regions consists of the union of a visible subregion on the display (2112) and an 
invisible subregion located outside the display (2112) and adjacent the visible subregion. For 
example, the selectable region at 1 1 o'clock in Figure 1 7 labeled with menu option "vort<space>x" 
5 consists of invisible subregion (0104) and visible subregion (0106), and within this description of 
the Perimeter Menu aspect of the invention is referred to as selectable region (0104/0106). The 
other selectable regions shown in Figure 17, proceeding counter clockwise from selectable region 
(0104/0106) are (0108/01 10), (01 12/01 14), (01 16/01 18), (0120/0122). (0124/0126), (0128/0130) 
and (0132/0134). Each subregion may be sized to suit the operator's preferences and abilities. 

10 Each selectable region is associated respectively with a menu option. In Figure 1 7, selectable 
region (0104/0106) is associated with menu option vort<space>x. selectable region (0108/01 10) 
with menu option "sumac", selectable region (01 12/01 14) with menu option "wizen"', selectable 
region (01 16/0118) with the menu option undo indicated by an icon on visible subregion (01 18) 
representing an undo function, selectable region (0120/0122) with menu option "words", selectable 

15 region (0124/0126) with menu option "talk", selectable region (0128/0130) with menu option 

"ldhbfk" and selectable region (0132/0134) with menu option "ypgqj,". Together, the eight visible 
subregions circumscribe region (0150) on the display. 

Selectable regions may be delimited by data indicative of one or more boundaries of the selectable 
20 region. Equivalently. the delimit means may be detectors operative to determine when the location 
indicated by the movement related signal has crossed one of those boundaries or intersects a 
selectable region. A partially delimited region or subregion is one which is unbounded on at least 
one side. 

25 Resuming, now, with the example, to select a menu option associated with a selectable region the 
operator moves a pointer (2202 in Figure 15) coupled to the general purpose computer system 
(2218 in Figure 15) to indicate a location on the selectable region, including either subregion, 
associated with the desired menu option and maintains the indicated location on the selectable 
region for the selection threshold period. The period of time required for selection may vary 

30 responsive to the proximity of the indicated location to the location of a cursor on the display or to 
the proximity of the indicated location to a point within the intersected selectable region. Dwell 
time may be continuous, discontinuous or dynamic (described below) for either or both subregions 
of the selectable region. 
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Selection in the above example is in response to a dwell event. A dwell event includes, but is not 
limited to, each of the following: (a) the durations of one or more periods of intersection of 
locations indicated by a movement related signal, a body member or a cursor (including any part of 
the cursor) and a selectable region equalling or exceeding a predetermined period; (b) a first 
5 quantity responsive to the durations of the periods referred to in (a) equalling or exceeding a 
predetermined quantity; (c) dwell event (a) or (b) followed by a location indicated by the 
movement related signal, the body member or the cursor no longer intersecting the intersected 
selectable region: and (d) dwell event (a) or (b) wherein the period of intersection required for 
selection of a selectable region increases in response to a non- intersection or a period of non- 
10 intersection of locations indicated by the movement related signal, the body member or the cursor 
and the selectable region ("dynamic dwell event"). The use of non-intersection or a period of non- 
intersection in determining the duration of a period of intersection required for selection is called 
dynamic dwell. Associated with each type of dwell event is an intersected selectable region. This 
is the selectable region intersected by the location indicated by the movement related signal, body 
15 member or cursor which triggers the dwell event by causing the period or the first quantity to equal 
or exceed the predetermined period or the predetermined quantity, respectively. 

Selection may also be in response to a selection event. A selection event includes, but is not 
limited to: (a) a dwell event; (b) a switch operation at or near the time of an intersection of a 

20 location indicated by a movement related signal, a body member or a cursor and a selectable 

region; (c) an intersection of a location indicated by a movement related signal, a body member or 
a cursor and a selectable region; and (d) selection event (c) followed by a location indicated by the 
movement related signal, the body member or the cursor no longer intersecting the selectable 
region it previously intersected. Associated with each type of selection event is an intersected 

25 selectable region. This is the selectable region intersected by the location indicated by the 

movement related signal, body member or cursor. The fact that a selection event has occurred may 
be indicated to the operator, for example, visually by changing the cursor appearance or location, 
by changing location, size, shape, hue, brightness, contrast, tone, dithering, pattern, hatching, font 
or fill of an object on the surface, or by displaying a graphic or a point distinguishable from its 

30 immediate surroundings on a surface or removing a graphic or point distinguishable from its 

immediate surroundings from a surface; auditively by generating a sound or changing the pitch or 
volume of an extant sound; tactilely by changing the surface or temperature of a contact area or the 
pressure exerted by a contact area: or by other means. In the prototype, following selection, the 
hue of the visible subregion of the selected selectable region is changed from green to magenta. 
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As used herein, a cursor includes a temporary marking on a display which emphasizes to an 
operator, in an optical manner, a momentarily important location or object. As used herein, body 
member means any part of the body including, but not limited to, each of the following: the 
shoulder, arm, elbow, wrist, hand, finger, thumb, leg. knee, ankle, foot, toe, hip, trunk, neck, 
5 tongue, lip, eye and head. The received movement related signal includes, but is not limited to, a 
signal indicative of movement or from which movement can be derived, such as a plurality of 
relative or absolute positions or a difference between two relative or absolute positions. Movement 
related signal receiving means includes, but is not limited to, each of the following: (a) pointer 
interface circuitry found in a general purpose computer system; (b) one or more detectors operative 
10 to detect movement of a pointer: and (c) one or more detectors operative to detect movement of a 
body member of an operator. In the prototype, the movement related signal receiving includes 
electronic circuitry in the general purpose computer system (221 8) operative to receive the 
movement related signal generated in part by the movement of the pointer (2202). 

15 In the prototype subregions are displayed on the display (2112). However, other means for 

displaying may be substituted for the means used in the prototype, for example, a projector for 
projecting an image, a surface having a static display thereon, or other suitable means. 

Resuming, now, with the example of Figure 17. and assuming that the operator has selected menu 
20 option "vort<space>x", the display is changed to that shown in Figure 1 8. In Figure 1 8, each of six 
selectable regions is now associated with a submenu option of the selected menu option 
"vort<space>x". Selectable region (0104/0106) is now associated with submenu option "<space>", 
selectable region (0108/01 10) with submenu option "o", selectable region (01 12/01 14) with 
submenu option "t", selectable region (0124/0126) with submenu option "x". selectable region 
25 (0128/0130) with submenu option "v", and selectable region (0132/0134) with submenu option "r". 
Selectable regions (0116/011 8) and (0120/0122) remains associated with the same menu options 
with which each was associated in Figure 1 7. The operator may now select one of these submenu 
options. 

30 Assuming that the selected submenu option is one of v, o. r, t, x and space, the selected character, 
or a corresponding computer encoding of that character, may be input to an apparatus coupled to 
the general purpose computer system (2218 in Figure 1 5). or input to an application program 
(1208) executing on the general purpose computer system (2218) coupled to the display (2112). 
Inputting, as used herein, includes, but is not limited to, generating or passing signals 
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representative of the selected menu option along a path toward the destination apparatus or 
program. Preferably, the computer program displays at least some of its output in the 
circumscribed region (0150). 

5 Given a display having eight selectable regions, an operator may. with a single selection indicate 
one of eight menu options, with two selections indicate one of up to 64 different menu options, 
w ith three selections indicate one of up to 256 menu options, etc. Each of these menu options may 
represent a sequence of one or more characters, a sequence of one or more data or control inputs to 
an application program (1208), or a control function for one or more devices or speech synthesizers 

10 coupled to the general purpose computer system (2218). As used herein, a character includes a 
space, a control character as defined by the American National Standards Institute (ANSI) or the 
American Standard Code for Information Exchange (ASCII), and a letter from one of the 
Afrikaans. Albanian. Amharic. Arabic, Armenian. Assamese, Assyrian, Avar, Azerbaijani, 
Balinese, Bamara, Bantu, Bashkir, Basque, Bengali. Birhari, Bulgarian, Buluba-Lulua, Burmese. 

15 Buryat, Byelorussian, Caddoan, Catalan. Chechen. Chikaranga, Chippewa. Choctaw, Church 

Slavik, Chuvash. Coptic, Cree, Croatian, Cyrillic, Czech. Dakota, Danish, Dari, Devanagari, Dutch, 
Dzongkha, English, Eskimo, Esperanto, Estonian, Ewe, Farsi, Fijian, Filipino, Finnish, Flemish, 
French, Fulani, Gaelic, Galician, Georgian, German, Greek, Gujarati, Gurmakhi, Harari, Hausa, 
Hawaiian, Hebrew, Hindi, Hiragana, Ibo, Icelandic, Indonesian, Irish, Irogquoian, Italian, 

20 Kabardian, Kalmyk, Kannada, Kanuri, Kashmiri. Katakana. Kazakh. Khasi. Khmer, Kirghiz, 
Kishmiri, Komi, Kongo, Kurdish, Lao, Latin, Latvian. Lithuanian, Lu-Guanda, Macedonian, 
Magahi, Maithili, Makua, Malagasy, Malay, Maiayalam, Maltese, Mandingo, Manipuri, Marathi, 
Masai. Mizo. Moldavian. Mongolian. Munda. Naga. Navaho. Nyanja. Nepalese. Norwegian. Oriya. 
Oromo, Ossetian. Pashto. Polish. Portugese, Punjabi, Rajasthani. Rhaeto-Romanic, Rumanian, 

25 Russian, Samoan. Sangs, Serbian, Serbo-Croatian, Sinhalese, Sinhi, Sioux, Slovak, Slovenia, 

Spanish, Sundanese, Swahili, Swedish, Syriac, Tadzhik, Tagalog,Tajik, Tamil, Tatar, Telugu, Thai, 
Tibetan, Turkish. Turkmen, Udmurt, Uighur, Ukranian. Umbundu, Urdu, Uzbek, Vietnamese, 
Visayan, Welsh, Yakut, Yoruba and phonetic alphabets. As used herein, each of a character, 
ideograph, control input and control function includes a computer encoding of the same. As used 

30 herein, a device includes, but is not limited to, each of a wheelchair, a household appliance, an 

appliance for use in an office, a workstation, a robot, and a computer peripheral. Thus, by selecting 
from a menu, the operator may. for example, increase the volume of an external speech synthesizer, 
or turn a wheelchair to the left. 
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The selectable regions organized as described above help an NMD operator make the menu 
selection he intends. Referring to Figure 17, suppose, for example, an NMD operator intends to 
move a pointer (2202) that is indicating point (0154) to indicate point (0156), a location in 
subregion (0130), but who is unable to quickly stop motion, so that the location indicated by the 
5 pointer (2202) moves from point (0154) past point (0156) to point (0158). Because point (0158) 
lies within the same selectable region (0128/0130) as the overshot subregion (0130), dwelling at 
point (0158) operates to select the intended selectable region (0128/0130). Invisible subregions. in 
accordance with the Perimeter Menu aspect of the invention, may extend outward from the edge of 
the display (2112) to infinity . In such an embodiment, dwelling at point (0162) would operate to 

10 select selectable region (0128/0130). Preferably, invisible subregions extend a finite distance from 
the edge of the display (21 12). In such an embodiment, dwelling at point (0162) would not operate 
to select selectable region (0128/0130). The sizes of the invisible subregions shown in Figures 17 
and 18 are illustrative only. Preferably the size of each invisible subregion is large enough to 
encompass overshoot but small enough to avoid unintentional selections when the NMD operator 

15 turns to see someone or something. 

The prototype utilizes the area outside the display to facilitate menu selection by a disabled 
operator. If an operator has impaired ability to maintain a steady position, he can point to a 
relatively large invisible subregion outside the display which is more forgiving of the operator's 

20 involuntary motion than the relatively small selectable regions on the display in conventional on- 
screen keyboards. Thus, the effective area of a selectable region is expanded beyond the region's 
visible subregion shown on a display. If an operator has impaired ability to stop motion he may, 
starting from the center of the screen, point to any selectable region. His impaired ability to stop 
will not impair his ability to select his intended target, assuming his directional control is relatively 

25 unimpaired, since, in the prototype, each selectable region is unbounded on its side furthest from 
the center of the display. Although the selectable regions of the prototype are large, only the 
visible subregion of each selectable region uses space on the display. Thus a large rectangular 
region remains available on the display for the output of an application program. The Perimeter 
Menu aspect of the invention is preferably implemented on a general purpose computer system. If 

30 the general purpose computer system is coupled to a speech synthesizer and the menu hierarchy 
allows the selection of letters and/or words, an operator having impaired speech may speak using 
the speech synthesizer. If a word processing or data entry application program is run on the 
general purpose computer system, the operator may enter words or data, respectively, for input to 
the application program. If the general purpose computer system is coupled to a devices capable of 
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executing commands and the menu hierarchy allows the selection of commands, a disabled 
operator may select and issue commands to control these devices. 



Since the cognitive demand for scanning is greater than that for direct selection, the prototype 
5 places less cognitive demand on the operator than a scanning system for selecting options from a 
menu. 

Figure 19 illustrates a display in accordance with an alternative embodiment of the Perimeter 
Menu aspect of the invention having eight selectable regions circumscribing a central region (6950) 
10 on the display. 

Figure 20 illustrates a display in accordance with an embodiment of the Perimeter Menu aspect of 
the invention having twenty selectable regions circumscribing a central region (0806) on the 
display. 

15 

Figure 2 1 illustrates a display in accordance with an embodiment of the Perimeter Menu aspect of 
the invention having four selectable regions (0508), (0506), (0504) and (0502) circumscribing a 
central region (05 10) on the display. 

20 Figure 22 illustrates a display in accordance with an embodiment of the Perimeter Menu aspect of 
the invention having four selectable regions (4008), (4006), (4004) and (4002) with no space 
between them other than a circumscribed region (4010) on the display. 

Figure 23 illustrates an apparatus in accordance with of another embodiment of the Perimeter 
25 Menu aspect of the invention. In Figure 23, region (3510) is located on interior display (3514) 
which is circumscribed by peripheral display (35 12). Selectable regions (3508), (3535), (3504) 
and (3502) are located on the peripheral display (3512). 

Figures 24 and 25 are each illustrations of a display and structures in accordance with another 
30 embodiment of the Perimeter Menu aspect of the invention. Figure 24 depicts ten selectable 

regions (6502), (6504), (6506), (6508), (6510), (6512), (6514), (6514), (6516), (6518), and (6520). 
Each selectable region is located on the display (2112) adjacent the edge of the display and 
associated respectively with a menu option. In Figure 24. the menu options are shown on their 
associated selectable region. Together the ten selectable region circumscribe region (6550) on the 
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display. In response to only an intersection of a location indicated by a movement related signal 
and selectable region (6506), the display changes to that shown in Figure 25, on which are located 
ten selectable regions each located on the display (2112) adjacent the edge of the display, nine of 
the ten selectable regions associated respectively with a submenu option. Selectable region (6608) 
is not associated with a submenu option. Any submenu option may be selected by a selection 
event. 

Figures 26 and 27 illustrate an apparatus in accordance with still another embodiment of the 
Perimeter Menu aspect of the invention. Figure 26 depicts a front view of the apparatus; Figure 27 
a cut away view from the top of the apparatus. Figure 26 depicts detector area (0348 in Figure 27) 
on, in or below which are located a plurality of selectable regions (0304), (0306), (0308), (0310), 
(0312), (0314), (03 16), (03 1 8), (0320), (0322), (0324), (0326), (0328), (0330), (0332), (0334), 
(0336), (0338), (0340) and (0342). Adjacent the detector area (0348 in Figure 27) is a berm (0350 
in both Figure 26 and 27) for confining a body member of the operator or a pointer controlled by 
the operator to the detector area (0348 in Figure 27). 

Still another apparatus in accordance with the Perimeter Menu aspect of the invention is illustrated 
in Figure 28 which depicts a headrest for an operator using his head to indicate a location on a 
display. Figure 28 shows an irregularity (0703) on the surface (0701) of the headrest. The 
irregularity is in physical contact with the operator and tactilely indicates to him the position of his 
head. The tactile indication means may be concave, convex or both or may differ from the surface 
in temperature. For individuals having impaired ability to sense the position of a body member, 
e.g. the operator's head, the tactile input thus provided to the operator improves the operator's 
ability to sense the position of his head. 

The prototype of the invention will now be described in detail and where the preferred mode of 
practicing the invention differs from the prototype, the preferred mode is described. The 
description is broken into several parts: 

1 . A brief overview of how a state table works. 

2. A description of the operation of the events, state table and state processing used in the 
prototype. 

3. A general description of each event and one example of the use of that event. 

4. An example of state machine processing in the operation of the prototype. 

47 

RECTIFIED SHEET ( RULE 91) 
ISA/EP 



WO 96/30822 PCTAJS95/03591 

The prototype implements the Perimeter Menu aspect of the invention as a state table. A state table 
is a tool for processing sequential inputs and is most easily understood by analogy. Imagine 
yourself in a room having a ticket window and three exits, each regulated by a turnstile. You 
collect a ticket at the ticket window which, when inserted into the appropriate turnstile, allows 

5 passage to a connecting room. The turnstile keeps the ticket. Any given ticket operates only one 
turnstile in a room, though different tickets may operate the same turnstile. You begin in a certain 
room, collect a ticket, insert it into the appropriate turnstile and pass to a connecting room, where 
vou perform certain tasks associated with the new room. Then you collect another ticket from the 
ticket window in that room, insert the ticket into the appropriate turnstile in that room, pass to a 

10 connecting room, perform certain tasks associated with the new room, and so on. 

The state table used in the prototype is depicted in Figures 29 and 30. Moving from the analogy 
above, the rooms are states represented by the rows of the state table; the tickets are events 
represented by the columns of the state table. Each entry in the state table represents a passage 
15 from one state to another. The tasks performed upon entry into a room correspond to the 

processing performed by the processor (2104) on entry to a new state ("state processing"). For 
example, state processing may cause a slight lightening of the color of a selectable region. Reentry 
into that state five times may successively lighten a selectable region five times. 

In the prototype, each selectable region is directly controlled by one associated state machine. 
Each state machine directly controls only its associated selectable region. Each state machine 
includes data uniquely associated with its associated selectable region, the shared state table shown 
in Figures 29 and 30, and the shared code for state processing, described below. This embodiment 
means that the state machine associated with selectable region (1614) may be in state 
STSELECTED while the state machine associated with selectable region (1602) is in state 
ST_INITIAL. In the prototype, these separate states are reflected only in separate values for data 
uniquely associated with each state machine. Each state machine has a unique index. 

The state table used in the prototype defines 18 states, composed of states zero through seventeen 
30 shown in Figures 29 and 30. State 1 (STERRORSTATE in Figure 29) is not used. Preferably, it 
is omitted. In the prototype, the differences between the several state machines, for example, the 
state of a particular state machine at any given time, are confined to data structures associated with 
that state machine. The state machines share the same code and the same state table. In other 
words, returning to the ticket and turnstile analogy, there are multiple travelers each with his own 
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ticket (event) and his own baggage (data) moving from room to room in the same labyrinth. Each 
traveler's actions in each room usually affect only his own baggage. 

In the prototype, there are two types of sequential inputs processed by the state table: external 
5 events and internal events. External events are generated outside the state table, for example, by 
the operator moving the pointer (2202) or by a timer expiring. Pointer movement may generate an 
event indicating that the operator has moved the cursor across the selectable region boundary from 
without the selectable region to within it. This event causes a transition from one state to another 
("drives" a state machine to a new state). For example, assuming a state machine is in state 
10 ST_CREST TIDE, row 6 in the state table shown in Figure 29. when an event EVCROSSOUT, 
column 4 of the state table, occurs. At the intersection of row 6 and column 4 is a 7. This 
represents the new state, row 7 of the state table, state STSELECTED. Thus event 
EV_CROSS_OUT drives the state machine from state STCRESTTIDE to state ST SELECTED. 
On entry to the new state, the computer performs the state processing associated with the new state. 

15 

Internal events are generated during state processing to handle circumstances where a first state 
transition is made due to an external event and the processing associated with the new state 
determines that a second state transition is necessary. The first, external, event has already been 
used so a second, internal, event is generated by the state. For example, if the operator has a 

20 prolonged muscle spasm, common among individuals with CP, or loses his grip on a hand held 

pointing device, the cursor may sit without moving on a selectable region for a considerable period 
of time. It is desirable to detect this condition, move the cursor to the center of the screen so the 
operator can easily find the cursor, and reset all selectable regions to their initial color. Detection 
is accomplished with a timer. On timer expiration, a state machine transitions to state STIDLE 

25 which centers the cursor. From there it is desirable to transition to state ST_RESET which, among 
other processing, initializes selectable region color. The transition from state ST IDLE to state 
ST RESET is driven by an internal event, generated by state ST IDLE state processing. 



Although the preferred embodiment uses only one state to perform certain state processing, that 
30 state processing may be equivalently performed in multiple states. Likewise, the state processing 
of multiple states of the described embodiment may be equivalently performed in a single state. 

Although the prototype uses a state table to control the flow of program execution and to select one 
of a plurality of selectable regions, the same function may be equivalently performed using object 
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oriented software architecture, if-then-else statements or a combination of these. In particular, in 
accord with object oriented software architecture, each state machine may be equivalently 
represented as an instantiation of a selectable region class for processing inputs affecting a 
particular selectable region. 

5 

Although the described embodiment uses a single processor, state table and code for state 
processing for all the selectable regions, each of these may be duplicated. Alternate embodiments 
may include processors, electronic circuitry , state tables or code for state processing used for 
processing fewer than all the selectable regions or used to process certain selectable regions at one 
10 time and other selectable regions at other times. 

In the prototype, each state machine processes events independently of all other state machines, 
though a state machine may send an event to another state machine. For example, when the 
operator selects a selectable region, the associated state machine sends the event EVRESET to all 
1 5 other state machines so that all selectable regions revert to their respective initial colors. 

A single operator action may result in the issuance of different events to different state machines. 
For example, when the operator moves the cursor from without a selectable region to within it, 
EV_DWELL is sent to the newly indicated state machine. EV_MOVEMENT is sent to all other 
20 state machines. 

Following is a general description of each event used in the prototype and an example of the use of 
each event. For all uses of each event, refer to the state table shown in Figures 29 and 30 which 
determines what state transition occurs from every state on occurrence of a given event. 

25 

The event EV RESET is an internal event which drives a state machine to its initial state. For 
example, when the operator has not moved the pointer (2202) for a predetermined period of time, 
event EV RESET is sent to all other selectable regions. 

30 The event EV DECAY is an external event which indicates that the cursor hotspot does not 

intersect the selectable region associated with the state machine. EV_DECAY is sent to a state 
machine periodically when the operator has positioned the cursor hotspot on a selectable region 
other than the selectable region associated with that state machine. 
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The event EV DWELL is an external event which indicates that the cursor hotspot intersects the 
selectable region associated with the state machine. EV DWELL is sent to a state machine 
periodically when the operator has positioned the cursor hotspot on the associated selectable 
region. 

5 

The event EV CROSS OUT is an external event which indicates that the cursor hotspot has moved 
from a location intersecting the selectable region associated with the state machine to a location not 
intersecting the selectable region. After the operator selects a selectable region, he must move the 
cursor hotspot out of the selectable region, generating EV CROSS OUT. before he can again 
1 0 select that selectable region. 

The event EVSTEPUP is an internal event which indicates that a selectable region's selection 
threshold has been satisfied. 

1 5 The event EVMOVEMENT is an external event which indicates that the cursor hotspot has 

moved. If the cursor hotspot intersects a selectable region without moving for a predetermined 
period of time, a timeout occurs, causing all state machines to transition to the reset state. 
EVMOVEMENT drives the state machine out of the reset state. 

20 The event EV_IDLE_TIMEOUT is an external event which indicates that the cursor hotspot has 
intersected a selectable region without moving for a predetermined period of time. 
EVIDLETIMEOUT causes the state machine to move the cursor hotspot to the center of the 
display. 

25 The event EVCEILING is an external event which indicates that the cursor hotspot intersects a 

selectable region and the color of the selectable region equals the selectable region color ceiling. If 
the locking feature is enabled, EVCEILING drives the state machine to the begin lock state where 
it displays the lock icon. 

30 The null event, EVNULL, is a multi-purpose internal event used in a variety of situations to drive 
a state machine to another state. For example, after a timeout has been detected, 
EV IDLE TIMEOUT is generated and sent to the appropriate state machine driving it to the idle 
state, the receiving state machine sends itself EV NULL in order to drive itself to the reset state. 
The use of EV NULL here allows states to be simpler and the reset state to be reused. 
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The prototype uses eight partially delimited selectable regions. In the description below, the 
portion of each selectable region shown on the display is referred to as the visible subregion of the 
selectable region. The portion of each selectable region outside the display is referred to as the 
invisible subregion of the selectable region. Because the software driver (1202) confines the cursor 
5 hotspot to the Windows® cursor clipping rectangle, a rectangle on the display slightly smaller in 
area than the display, the access program (1206) of the prototype only reads hotspot cursor 
locations within the Windows® cursor clipping rectangle, even though the operator may in fact be 
pointing to a location outside the Windows® cursor clipping rectangle, e.g. within an invisible 
subregion. Thus the access program (1206) does not distinguish between two locations indicated 

10 by the operator, the first at a first location on the edge of the Windows® cursor clipping rectangle 
and the second outside the Windows® cursor clipping rectangle whose location is reported by the 
software driver (1202) to be the first location. For example, assuming that the operator moves the 
location indicated by the pointer (2202) to a location within invisible subregion (0104 in Figure 
17), the software driver (1202) in the prototype reports the cursor hotspot location to be the closest 

15 point within the visible subregion (0106). Consequently, in the prototype, all invisible subregions 
are unbounded on their side furthest from and parallel to the edge of the display. Thus, in the 
prototype, point (0162) in Figure 17 lies within selectable region (0128/0130) since the rightmost 
side of selectable region (0128/0130) is unbounded. 

20 The operation of the prototype will now be described using, as an example, the selection of a menu 
option associated with selectable region (0104/0106). First described are notation conventions 
used in the description, then initialization in the prototype, and then the example. The description 
refers to the procedures PocketFsm and CreateEvent. These procedures are listed in pseudo-code 
in Appendix I. 

25 

Notational conventions used in the description below: 

1. pPocket-> indicates a set of data associated with a particular state machine, in this example, the 
state machine associated with selectable region (0104/0106). 

2. pPocket->State indicates a particular item of data within the set of data associated with the 
30 state machine, in this case, the variable "State". 



During initialization: 

1 . The Windows® cursor clipping rectangle is set so that most of the arrow cursor is always 
visible on the display. 
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2. The cursor, indicating on the display (2112) the location indicated by the movement related 
signal receiving means, is positioned at the center of the display. 

3. All data associated with each state machine are initialized. The following variables are 
included in the set of data associated with each state machine. There are as many independent 
copies of these variables as there are state machines. For each state machine the following 
variables are initialized as indicated. 



variable 
flnvert 



State 

PreviousState 

Color 

fPaint 

InitialColor 
pLabel 



Decrement 

Increment 

Ceiling 

CrestTide 

Lockspot.x 



Lockspot.y 



initial value 
FALSE 



STJNITIAL 

0 

0 

FALSE 

RGB (0,32,0) 
initial menu option 
for the state machine's 
selectable region 
RGB (0,1,0) 
RGB (0,4,0) 
RGB (0,255,0) 

RGB (0,143,0) 

a point within the 
selectable region 
located two thirds of 
the length of the 
selectable region from 
the closest corner of 
the display 
a point within the 
selectable region 



meaning 

if TRUE, display the color complementary to that 
indicated by this selectable region's Color 
variable 

state of the state machine 

previous state of the state machine 

RGB encoding of visible subregion color 

if TRUE, paint visible subregion on receipt of 

WM_PArNT message 

initial value corresponds to a dark green 

indicates an element within the 

aLabel array, described below 

value subtracted from Color on EVDECAY 
value added to Color on EV D WELL 
initial value corresponds to a very light, bright 
green 

initial value corresponds to a 
light green 

x coordinate of location for the 
display of the lock icon 



y coordinate of location for the 
display of the lock icon 
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located two thirds of 
the length of the 
selectable region from 
the closest corner of 
the display 

Anterior TRUE for interior if TRUE, this is an interior region 

regions, otherwise of adjacent regions 

FALSE: used for 
the Intersection 
aspect of the invention 

iAdjacentPocket 

index of state machine associated 

with the adjacent region: used for 

the Intersection 

aspect of the invention 
first array of points 

boundaries of the 

visible subregion 
second array of points 

boundaries of the 

region; used for the Intersection 

aspect of the invention 
hRegion handle to Windows® region 

corresponding to the visible subregion; 

used for the Intersection 

aspect of the invention 

4. aLabel, an array of data structures defining the menu and submenu options and the menu 
hierarchy, is initialized. For example, one of the elements of aLabel defines menu option 
"vort<space>x". This element includes fields which determine that this menu option is 
displayed horizontally starting at certain (x,y) coordinates, that on selection certain actions 
are to be taken, for example, outputting text to a speech synthesizer, and that on selection 
certain submenu options, in this example, "v", "o", "r", "t". "<space>" and "x". and related 
data are to be associated with certain state machines. In the prototype, this association is 
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accomplished by modifying pLabel in the set of data of the associated state machine to point 
to the aLabel element corresponding to the menu option to be associated with that state 
machine. 

5. At least one window is created in the circumscribed region (0150) for the display of selected 
letters. 

6. The state table is initialized to the values shown in Figures 29 and 30. 

7. EVRESET is sent to each state machine by a procedure call of the form PocketFsm 
(pPocket, EV RESET). As an example, assume pPocket indicates the state machine 
associated with selectable region (0104/0106). Proceeding through the pseudo-code for the 
procedure PocketFsm listed in Appendix I, flnternalEvent is set to TRUE and consequently 
control passes into the while loop. flnternalEvent is now set to FALSE. The current state, 
pPocket-> State, is stored in pPocket->PreviousState. Now a state transition is made. The 
current state. STFNITIAL. having a value of 2, and the current event. EV RESET, having a 
value of 1, are used as row and column indices respectively into the state table aPocketFsm 
shown in Figures 29 and 30. to determine the value of the new state of the state machine, in 
this example, the state machine associated with selectable region (0104/0106). 
aPocketFsm[2][l] equals 3. Thus the new state of the state machine is 3, the value of 
ST_RESET. Control passes, via the switch statement, to the ST_RESET case and 
ST_RESET state processing is performed. The time of selectable region selection is set to 
the current time, the state machine's flnvert flag is set to FALSE and the value of the state 
machine's color variable, pPocket->Color, is compared to the state machine's initial color, 
pPocket->InitialColor. pPocket->Color was initialized to zero, which is not the value of 
pPocket->lnitialColor. Consequently, pPocket->Color is set to pPocket->InitialColor and 
the flag pPocket->fPaint is set to TRUE. Upon reaching the break statement, control passes 
through the bottom of the switch statement and the value of pPocket->fPaint is tested. Since 
it is TRUE, the client area rectangle is invalidated. In the prototype, Windows® ( 1 204) 
responds to invalidating the client rectangle by sending the access program (1206) a 
WM_PAiNT message. On receipt of a WM_PAINT message, the access program (1206) 
redraws all selectable regions and any menu option located thereon for each state machines 
having pPocket->fPaint equal to TRUE. Thus, somewhat indirectly, visible subregion (0106) 
is drawn on the display (2112). Then control returns to Windows® (1204). The other seven 
state machines, each associated respectively with a selectable region, are similarly initialized 
so that each state machine transitions from ST INIT1AL to ST_RESET and draws its 
respective visible subregion and any menu option located thereon on the display (21 12). The 
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display shown in Figure 1 7 now appears on the display (2112) of the computer system 
(2116). 

8. A periodic timer, called the cursor polling timer, is set. This timer provides the access 
program (1206) with a WMTIMER message at frequent intervals, in the prototype every 

5 system clock tick which occurs approximately every 54 milliseconds. Following access 

program (1206) initialization, most state transitions are made on expiration of the cursor 
polling timer. The access program (1206) calls the procedure CreateEvent to determine the 
appropriate event for each state machine and to complete the event data structure addressed 
by pEvent accordingly, then repetitively calls the procedure PocketFsm for each state 

10 machine, passing the unique indicator for the state machine and the appropriate event for that 

state machine. Preferably, the cursor polling timer is more frequent so that color changes to 
visible subregions are smaller and more frequent, giving a smoother appearance to color 
change. 

1 5 An example of the selection of selectable region (0 1 04/0 1 06) in accord with the Perimeter Menu 
aspect of the invention will now be described. Following initialization, assume that the operator 
now begins to move the pointer (2202). Every 54 milliseconds the cursor polling timer expires, 
causing Windows® (1204) to send a WM_TIMER message to the access program (1206). 
Following receipt of WM_TIMER, the access program (1206) calls the procedure CreateEvent. 

20 The procedure CreateEvent, among other functions, determines whether the current cursor hotspot 
location lies within any selectable region. In this example, the operator is moving the cursor from 
its initial location in the center of the display toward selectable region (0104/0106), but since only 
54 milliseconds have elapsed, the cursor hotspot has moved only slightly in that direction. The 
procedure CreateEvent determines that the cursor hotspot does not lie within any selectable region 

25 and that the cursor hotspot has not crossed out of a selectable region in the past 54 milliseconds. 
Therefore, the procedure CreateEvent determines that each state machine should receive 
EV_MOVEMENT. The procedure PocketFsm is called with the indicator for the state machine 
associated with selectable region (0104/0106) and with EV MOVEMENT. The event 
EVMOVEMENT drives this state machine from its current state. STRESET, to STEBBTIDE. 

30 The pseudo-code for ST_EBB_TIDE in procedure PocketFsm is a break statement, indicating that 
no state specific action is taken at this time, other than the transition to ST EBB TIDE. Control 
returns to Windows® (1204). 

Shortly before or shortly after the state machine associated with selectable region (0104/0106) 
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receives EVMOVEMENT, all other state machines each receive EVMOVEMENT and each 
makes the same transition from STRESET to STEBBTIDE 

Another 54 milliseconds elapses and again procedure PocketFsm is called, sending 
5 EV MOVEMENT to the state machine associated with selectable region (0104/0106) and driving 
state machine from STEBBTIDE to STDECAY. Stepping through the pseudo-code for 
STDECAY. the ST DECAY state sets pPocket->State to the value stored in 
pPocket->PreviousState and decrements pPocket->Color but not below the value of 
pPocket->InitialColor. pPocket->Color determines the color and brightness of visible subregion 

10 (0106). Decrementing pPocket->Color results in a darkening of visible subregion (0106). 

Resuming with the pseudo-code for ST DECAY state processing, the ST DECAY state sets 
flntemalEvent to TRUE, and, in this case, sets Event to EVNULL. Following the break 
statement, the while flntemalEvent condition is true and another state transition occurs, using the 
value of pPocket->State which was set by ST DECAY state processing to ST EBB TIDE. the 

15 previous state. The new state is found at aPocketFsm [STEB BTI DE] [E V_NULL] , which equals 
ST EBB TIDE. This state transition is unlike an ordinary state transition because the starting state 
is set by ST_DECAY. All transitions from ST_DECAY share this distinction. The state machine 
executes the code for the new state, ST EBB TIDE, which is simply a break statement. The 
procedure PocketFsm determines that fPaint is FALSE and exits. Control returns to Windows® 

20 (1204). 

Shortly before or shortly after the state machine associated with selectable region (0104/0106) 
receives EV_MOVEMENT, all other state machines each receive EV MOVEMENT and each 
makes the same transitions from ST EBB TIDE to ST DECAY to ST EBB TIDE. 

25 

Every 54 milliseconds this scenario is repeated for each state machine until the operator moves the 
cursor hotspot to a point within selectable region (0104/0106). At this time the procedure 
CreateEvent determines that there is an active selectable region, specifically selectable region 
(0104/0106), and that consequently EV DWELL should be sent to the associated state machine. 
30 The procedure PocketFsm is called with the indicator for the state machine associated with 

selectable region (0104/0106) and the event EV DWELL. EV DWELL drives this state machine 
from ST_EBB_TIDE to ST ENTRY. Following the pseudo-code for STENTRY state processing 
shown in PocketFsm pseudo-code. flntemalEvent is set to TRUE and Event is set to EV NULL. 
resulting in another state transition to aPocketFsm[ST_ENTRY][EV_NULL]. which equals 
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ST_LOW_TIDE. 

The pseudo-code for STLOWTIDE is only a break statement, so there is no state specific action 
for STLOWTIDE other than entry into this state. fPaint is FALSE so the selectable region is not 
5 redrawn. Control returns to Windows® (1204). 

Shortly before or shortly after the state machine associated with selectable region (0104/0106) 
receives EVDWELL. all other state machines each receive EVMOVEMENT and each makes the 
same transitions from STEBBTIDE to ST DECAY to STEBBTIDE. 

10 

Another 54 milliseconds elapses. The procedure CreateEvent determines that the state machine 
associated with selectable region (0104/0106) should receive EV_MOVEMENT, which drives it 
from ST_LOW_TIDE to STDWELL. Stepping through the pseudo-code for STDWELL. the 
STDWELL state sets pPocket->State to the value stored in pPocket->PreviousState, increments 

15 pPocket->Color by pPocket->Increment, but not above the value of pPocket->Ceiling, sets 
pPocket->fPaint to TRUE, sets flnternalEvent to TRUE, and, in this case, sets Event to 
EV_NULL. Following the break statement, the while flnternalEvent condition is true and another 
state transition occurs. The new state is found at aPocketFsm[ST_LOW_TIDE][EV_NULL], 
which equals ST_LOW_TIDE. This path is unlike an ordinary state transition because the starting 

20 state is set by ST_DWELL. All transitions from ST_DWELL share this distinction. The state 

machine executes the code for the new state, ST LOW TIDE, which is simply a break statement. 
The procedure PocketFsm determines that fPaint is TRUE, invalidates the client rectangle and 
exits. As a result of invalidating the client rectangle, Windows® (1204) sends a WM PAINT 
message to the access program (1206). On receipt of WMPAfNT. the access program (1204) 

25 checks the value of fPaint for each state machine, and if TRUE, sets fPaint to FALSE and redraws 
the visible subregion of the selectable region associated with that state machine and any menu 
option located thereon. The color of the redrawn visible subregion is determined by the value of 
the Color variable for that state machine. After redrawing, control returns to Windows® (1204). 

30 Shortly before or shortly after the state machine associated with selectable region (0104/0106) 

receives EVJDWELL, all other state machines each receive EV_MOVEMENT and each makes the 
state transitions from ST_EBB_TIDE to ST_DECAY to ST EBB TIDE. 

Assuming the operator maintains the cursor hotspot in the selectable region (0104/0106), the state 
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machine associated with selectable region (0104/0106) cycles repetitively through the state 
transitions from STLOWTIDE to STDWELL to STLOWTIDE, driven by the cursor polling 
timer. With each transition to ST DWELL, visible subregion (0106) is brightened a bit. The 
polling timer interval is short enough and the increment to pPocket->Color is small enough that 
5 visible subregion (0106) appears to gradually brighten although in fact it progresses rapidly 
through a series of discrete brightness levels. With each iteration through STDWELL, 
pPocket->Color in incremented. 

Assuming that the operator maintains the cursor hotspot on selectable region (0104/0106), 

10 pPocket->Color will eventually equal or exceed pPocket->CrestTide, a variable set at initialization 
time and not changed thereafter. At this time, Event is set to EV_STEP_UP, driving a transition to 
aPocketFsm[ST_LOW_TIDE][EV_STEP_UP], which equals STSELECTED. The state 
processing for ST SELECTED provides the operator with an audible indication that a selection has 
just been made, takes the action appropriate upon selection of this selectable region, including 

15 selecting the menu option associated with the selected selectable region. In this example, the 
access program (1206) does not generate output to another program or device at this time. 
ST_SELECTED next sets pPocket->flnvert to TRUE, and, if appropriate, changes the menu 
options associated with various selectable regions. In this example, selectable region (0104/0106) 
is now associated with menu option "<space>", selectable region (0108/01 10) with menu option 

20 "o", selectable region (01 12/01 14) with menu option "t", selectable region (0124/0126) with menu 
option "x", selectable region (0128/0130) with menu option "v", and selectable region (0132/0134) 
with menu option V. Selectable regions (01 16/01 18) and (0120/0122) remains associated with the 
same menu options with which they were associated in Figure 1 7. fPaint is set to TRUE for the 
state machines associated with the selectable regions having changed menu options. 

25 ST SELECTED next sets Event to EV NULL and flntemalEvent to TRUE. The state machine 
now makes the transition to aPocketFsm[ST_SELECTED_TIDE][EV NULL], which equals 
ST_CREST_TIDE. There is no state specific action for ST CREST TIDE other than entry into 
this state. The pseudo-code for ST_CREST_TIDE is only a break statement. Control returns to 
Windows® (1204). 

30 

When the access program (1206) later receives the WM PAINT message resulting from the 
invalidating of the client rectangle, the setting of flnvert to TRUE causes the visible subregion 
(0106) to be drawn in the color complementary to the value then indicated by pPocket->Color. 
The setting of fPaint to TRUE for all selectable regions associated with changed menu options 
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causes those selectable regions and the menu options thereon to be redrawn. The display shown in 
Figure 18 now appears on the display (2112) of the computer system (21 16). 

Assuming the operator maintains the cursor hotspot in the selectable region (0104/0106), the state 
5 machine associated with selectable region (0104/0106) cycles repetitively through the state 

transitions from STCRESTTIDE to ST DWELL to STCRESTTIDE, driven by the cursor 
polling timer. With each transition to ST DWELL, visible subregion (0106) is brightened a bit. 
though it is now magenta, the complement of green. The polling timer interval is short enough and 
the increment to pPocket->Color is small enough that visible subregion (0106) appears to 
10 gradually brighten although in fact it progresses rapidly through a series of discrete brightness 

levels. With each iteration through ST_DWELL, pPocket->Color in incremented. Assuming that 
the operator maintains the cursor hotspot on selectable region (0104/0106), pPocket->Color will 
eventually equal pPocket->Ceiling. a variable set at initialization time and not changed thereafter. 

15 Assuming that the operator now moves the cursor so that the hotspot is located outside selectable 
region (0104/0106), CreateEvent generates the EV_CROSS_OUT for the state machine associated 
with selectable region (0104/0106), driving the state machine to 

aPocketFsm[ST_CREST_TIDE][EV_CROSS_OUT], which equals ST_SELECT_AND_OUT. 
The state processing for ST_SELECT_AND_OUT sends EV RESET and then EV_MOVEMENT 

20 to all state machines, driving each of them from their current state to STRESET and then to 

ST_EBB_TIDE. For each state machine, the transition ST RESET and then to ST EBB TIDE 
results in actions previously described for those states, except that, unlike before, the value of 
pPocket->Color for the state machine associated with selectable region (0104/0106) does not equal 
the value of pPocket->InitialColor for the state machine associated with selectable region 

25 (0104/0106). Consequently, pPocket->Color is set to pPocket->InitialColor and fPaint is set to 
TRUE, subsequently causing visible subregion (0106) to be redrawn in its initial color. Control 
returns to Windows® (1204). 

The prototype continues to sample cursor location at 54 millisecond intervals, determine the 
30 appropriate event for each state machine and send that event each state machine, causing state 
transitions in each state machine according to the state table shown in Figures 29 and 30. 
Assuming that the operator next selects selectable region (0108/01 10). the state processing in 
ST S ELECTED displays the selection, the letter "o". in the circumscribed region (0150), and 
associates the menu options shown in Figure 17 with their respective selectable regions. The 
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display shown in Figure 17 appears on the display (2112) of the computer system (21 16). 
B. Confinement 

5 The preferred embodiment of the Confinement aspect of the invention will now be described in 
detail from a functional perspective using an example illustrated in Figure 3 1 . In Figure 3 1 , 20 
selectable regions, e.g. (1704), (1708) and (1714), are depicted on display (2112). Each of the 
selectable regions is located on the display (21 12) adjacent an edge of the display, and the 
selectable regions together circumscribe a region (1702) on the display. The top and left edges of 

10 the Windows® cursor clipping rectangle (1750) lie on the top and left edges, respectively, of the 
display (2112). The bottom and right edges of the Windows® cursor clipping rectangle ( 1 750) lie 
on the display parallel to and slightly indented from the bottom and right edges, respectively, of the 
display. A confining polygon is delimited on the display. The boundary of the confining polygon, 
starting from the upper right corner of the Windows® cursor clipping rectangle, follows the top 

15 edge of the Windows® cursor clipping rectangle to the left until it reaches region (1732), where the 
boundary follows the side of region ( 1 732) down, to the left, and back up to the top edge of the 
Windows® cursor clipping rectangle. The boundary continues left along the top edge of the 
Windows® cursor clipping rectangle until it reaches region (1 734), where the boundary follows the 
side of region ( 1 734) down, to the left, and back up to the top edge of the Windows® cursor 

20 clipping rectangle. The boundary continues left along the top edge of the Windows® cursor 
clipping rectangle to the upper left comer and then turns down along the left edge of the 
Windows® cursor clipping rectangle until it reaches region (1 736), where the boundary follows the 
side of region ( 1 736) to the right, down, and to the left to the left edge of the Windows® cursor 
clipping rectangle. The boundary of the confining polygon continues in this fashion around to the 

25 upper right corner of the Windows® cursor clipping rectangle. The confining polygon thus 

includes all the area of the Windows® cursor clipping rectangle except for the regions (1732), 
( 1 734), ( 1 736), ( 1 73 8), ( 1 740), ( 1 742), ( 1 744), and ( 1 746). An operator controlling a pointer 
indicating successive locations with respect to the display and attempting to select a target 
selectable region may overshoot the target so that some of the successive locations lie outside the 

30 Windows® cursor clipping rectangle. In the preferred embodiment of the Confinement aspect of 
the invention, the cursor (1724) is confined to the confining polygon. The preferred embodiment is 
responsive to an intersection of the cursor hotspot and any one selectable region so that an overshot 
selectable region may be selected by click or by dwell without moving the location presently 
indicated by the pointer to a location in the overshot selectable region. Thus, the preferred 
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embodiment of the Confinement aspect of the invention allows an operator with impaired ability to 
stop motion to maintain the cursor more easily on a selectable region, and so select the intended 
selectable region, than do conventional user interface systems. 

5 A selectable region having a single side abutting a confining polygon prevents cursor movement 
only beyond the abutting side. However. NMD operators who drift may drift in more than one 
direction. Assume that a certain NMD operator tends to drift both up and to the left and that he is 
attempting to select selectable region (1704). If he moves the cursor into that selectable region his 
upward drift will be confined: the drift will not move the cursor beyond the confining polygon. 

10 However, the cursor will move to the left, since, in the preferred embodiment of the Confinement 
aspect of the invention, movement in this direction is not affected by the confining polygon, and 
consequently the cursor may move into selectable region (1708), the selectable region to the left of 
selectable region (1 704). NMD operators having this type of drift may be assisted in selecting by 
confining comers. For example, such an operator, attempting to select selectable region ( 1 704) 

15 could move the cursor to location (1706) in selectable region (1704). As the operator drifts to the 
left, he can compensate by moving the pointer to the right. Assuming the operator lacks fine motor 
control, he may overcompensate and indicate a location to the right of the Windows® cursor 
clipping rectangle ( 1 750). However, since the cursor is confined to the confining polygon, the 
cursor remains in the intended selectable region. 

20 

Confining corners facilitate the selection process for some NMD operators. The preferred 
embodiment of the Confinement aspect of the invention creates a corner or virtual comer in each 
selectable region. A virtual comer is a comer of a selectable region formed by the intersection of 
two sides of a selectable region both of which abut a confining polygon. For example, comer 

25 (1710) in selectable region (1708) abuts the confining polygon both along the top edge of the 
Windows® cursor clipping rectangle and along the right side of region (1732). If an NMD 
operator drifts from selectable region ( 1 708) to the left into region ( 1 732), the cursor remains in 
selectable region (1708). Thus, drift to the left does not move the cursor out of selectable region 
(1732). An operator trying to select selectable region (1 708) may overcompensate for drift to the 

30 right by moving the pointer to indicate a location in region (1732). 

Figure 32 illustrates a display in accordance with an alternative embodiment of the Confinement 
aspect of the invention. Figure 32 depicts 16 selectable regions, e.g. selectable region (0602), on a 
display (2112), the selectable regions together at least partially circumscribing region (0660) on the 



62 



WO 96/30822 



PCTAJS95/03591 



display. Circumscribed region (0660) intersects four selectable regions, (0604), (0608), (0610), 
and (0612). Selectable region (0608) includes virtual corner (0652). The confining polygon 
includes all the area of the Windows® cursor clipping rectangle except for regions (0632), (0634), 
(0636), (0638), (0640), (0642), (0644), and (0646). 

5 

Figure 33 illustrates a display and structures in accordance with an another embodiment of the 
Confinement aspect of the invention. Figure 33 depicts 20 selectable regions, e.g. selectable region 
(0918), on a display (21 12). the selectable regions together at least partially circumscribing region 
(0902) on the display The confining polygon includes all the area of the Windows® cursor 
10 clipping rectangle except for regions (0906), (0908), (0910), and (0904). 

The preferred embodiment of the Confinement aspect of the invention will now be described in 
detail from an implementation perspective. Preferably, the Confinement aspect is implemented by 
modifications to the access program (1206) described in the detailed description of the Perimeter 

1 5 Menu aspect of the invention. The modifications required are: ( 1 ) define 20 state machines and 20 
respectively associated selectable regions located as shown in Figure 3 1 ; (2) at initialization time, 
create a Windows® region corresponding to the confining polygon described above in the 
functional description of the preferred embodiment of the Confinement aspect of the invention; and 
(3) modify the procedure CreateEvent so that before generating an EVCROSSOUT event, 

20 CreateEvent determines whether the current hotspot cursor location intersects the confining 
polygon, and, if not, set the Windows® hotspot cursor location to the previous hotspot cursor 
location and transfers control to the code at the beginning of the CreateEvent procedure which gets 
the current cursor hotspot location from Windows®. If the current hotspot cursor location 
intersects the confining polygon, CreateEvent takes the same action as in the access program 

25 (1206). 

C. Dwell 

The preferred embodiment of the Dwell aspect of the invention will now be described in detail 
30 from a functional and implementation perspective. The prototype implements dynamic dwell. The 
effect of dwelling on a selectable region and its implementation have been described in the detailed 
description of the Perimeter Menu aspect of the invention. Now the effect and implementation of 
moving the cursor hotspot off a selectable region will be described. 
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Referring now to Figure 1 7, assuming that the state machine associated with the selectable region 
(0104/0106) is in state ST_LOW_TIDE, that pPocket->Color has been incremented above its initial 
value, that the cursor hotspot intersected selectable region (0104/0106) at the last expiration of the 
cursor polling timer, and that the operator initiates movement of the cursor hotspot to the right 

5 from selectable region (0104/0106) toward selectable region (0132/0134) so that, when the cursor 
polling timer next expires, the cursor hotspot is located in area (0150) between visible subregions 
(0106) and (0134), the procedure CreateEvent determines that the state machine associated with 
selectable region (0104/0106) should receive EV CROSS OUT. which drives it from 
ST_LOW_TIDE to STLOWT1DE. The state processing for ST_LOW_TIDE has already been 

10 described in the detailed description of the Perimeter Menu aspect of the invention. 

Assuming that the operator continues to move the cursor hotspot toward selectable region 
(0132/0134). on the next expiration of the cursor polling timer, the procedure CreateEvent 
determines that the state machine associated with selectable region (0104/0106) should receive 

15 EV MOVEMENT, which drives it from ST_LOW_TIDE to STDECAY. Stepping through the 
pseudo-code for ST DECAY, the ST DECAY state sets pPocket->State to the value stored in 
pPocket->PreviousState, decrements pPocket->Color by pPocket->Decrement, but not below the 
value of pPocket->lnitialColor, sets pPocket->fPaint to TRUE, sets flnternalEvent to TRUE, and, 
in this case, sets Event to EV_NULL. Following the break statement, the while flnternalEvent 

20 condition is true and another state transition occurs. The new state is found at 

aPocketFsm[ST_LOW_TIDE][EV_NULL], which equals ST_LOW_TIDE. This path is unlike an 
ordinary state transition because the starting state is set by ST DECAY. All transitions from 
ST DECAY share this distinction. The state machine executes the code for the new state, 
ST LOW TIDE. which is simply a break statement. The procedure PocketFsm determines that 

25 fPaint is TRUE, invalidates the client rectangle and exits. As a result of invalidating the client 
rectangle, Windows® (1204) sends a WMPAINT message to the access program (1206). On 
receipt of WM PAINT, the access program (1204) checks the value of fPaint for each state 
machine, and if TRUE, sets fPaint to FALSE and redraws the visible subregion of the selectable 
region associated with that state machine and any menu option located thereon. The color of the 

30 redrawn visible subregion is determined by the value of the Color variable for that state machine. 
After redrawing, control returns to Windows® (1204). 

If the movement of the cursor hotspot pause between successive samplings of its location, the 
procedure Create Event will determine that not EV_MOVEMENT, but EV DECAY, should be 
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sent to the state machine associated with selectable region (0104/0106). Like EVDECAY, 
EVMOVEMENT drives the state machine associated with selectable region (0104/0106) to 
STDECAY. The same state processing as described above for STDECAY takes place, including 
the transition back to ST_LOW TIDE. 

5 

Assuming the operator maintains the cursor hotspot in area (01 50) . the state machine associated 
with selectable region (0104/0106) cycles repetitively through the state transitions from 
ST_LOW_TIDE to ST DECAY to ST_LOW_TIDE, driven by the cursor polling timer. With 
each transition to ST DECAY. visible subregion (0106) is darkened a bit. The polling timer 
10 interval is short enough and the decrement to pPocket->Color is small enough that visible 

subregion (0106) appears to gradually darken although in fact it progresses rapidly through a series 
of discrete brightness levels. With each iteration through ST_DECAY, pPocket->Color in 
decremented until pPocket->Color reaches pPocket->InitialColor. 

15 When the cursor hotspot reaches selectable region (0132/0134), the procedure CreateEvent 

determines that the state machine associated with selectable region (0104/0106) should receive 
EV_DECAY, continuing the repetitively cycling through from ST_LOW_TIDE to ST_DECAY to 
ST_LOW_TIDE, driven by the cursor polling timer. Driven by the same polling timer, the 
procedure CreateEvent determines that the state machine associated with selectable region 

20 (0132/0134) should receive EVDWELL, driving this state machine through the state transitions 
described in the detailed description of the Perimeter Menu aspect of the invention. 

If the operator moves the cursor hotspot from selectable region (0132/0134) back to selectable 
region (0104/0106), the procedure CreateEvent determines that the state machine associated with 
25 selectable region (0104/0106) should receive EV DWELL, driving this state machine through the 
state transitions described in the detailed description of the Perimeter Menu aspect of the invention 

In the prototype, the selectable regions are shown on a display (21 12). Alternatively, the selectable 
regions may appear on a static display, or they may be projected on a surface. 

30 

In the prototype, the brightness of a visible subregion at any time indicates the progress of the 
selection of the selectable region including the visible subregion. A brightness close to the initial 
brightness indicates that a relatively long period of dwelling on this selectable region is required 
for selection. A brightness close to the brightness just prior to selection indicates that relatively 
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short period of dwelling on this selectable region is required for selection. However, means for 
indicating an intersection of the location indicated by the movement related signal and a selectable 
region, or the duration of a period of such an intersection includes, but is not limited to. a change in 
cursor appearance or location, a change in location, size, shape, hue, brightness, contrast, tone, 
5 dithering, pattern, hatching, font or fill of an object on the surface, a display of or change in a 
graphic on the surface or the removal of a graphic from the surface, a generation of a sound or a 
change in the pitch or volume of an extant sound, a change in the temperature or surface of a 
contact area, the pressure exerted by a contact area, or frequency of contact by a contact area, or 
other suitable means. Any of these indications may be continuous or frequent. 

10 

Although dwell is implemented in the prototype using a data value, specifically pPocket->Color, it 
may be implemented using a signal, for example, voltage or current, varying in response to the 
intersection and subsequent non-intersection of a location indicated by a pointer and a selectable 
region. For example, a selectable region may include a detector and coupled electronics or 
15 electrical circuitry operative to increase the voltage level of a capacitor. Once elevated, the voltage 
level may decrease over time. Upon reaching a predetermined threshold, the voltage level may 
trigger selection. 

The prototype allows an operator to make selections by dwell more efficiently than in conventional 
20 systems. In the prototype, the brightness of a visible subregion indicates the dwell time required 
for selection. A practiced operator may accurately estimate when he may plan his next pointer 
movement, when he may begin moving the pointer and may determine when a bit more exertion 
will select a selectable region and when it will not. Thus a disabled operator who is fatigued by 
computer access and can maintain a pointer in a steady position for only brief periods, may 
25 optimize his energy expenditure, for example, exerting himself to maintain the cursor on a certain 
selectable region only when doing so will quickly select the selectable region. 

The prototype may increase the independence of a disable individual by allowing him to control 
devices such as a TV, thermostat and other household appliances. As stated earlier, the Perimeter 
30 Menu aspect of the invention may be implemented on a general purpose computer system. If the 
general purpose computer system is coupled to a devices capable of executing commands and the 
menu hierarchy allows the selection of commands, a disabled operator may select and issue 
commands to control these devices. 
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Figure 34 illustrates a display in accordance with of an embodiment of the Dwell aspect of the 
invention having 20 selectable regions (3101), (3103), (3105), (3107), (3109), (3111), (3115), 
(3 1 1 7), (3 1 1 9), (3 1 2 1 ), (3 1 25), (3127), (3 129), (3 1 3 1), (3 133), (3 1 35), (3 1 39), (3 14 1 ), (3 1 43), and 
(3 1 45) circumscribing a surface (3151) having an indicating region (3 1 47) thereon. 

5 

Figure 35 illustrates an apparatus in accordance with of an embodiment of the Dwell aspect of the 
invention having 20 selectable regions circumscribing a surface, each selectable region associated 
respectively with an indicating region adjacent its associated selectable region. Selectable region 
(3101) is associated with indicating region (3201). selectable region (3103) with indicating region 
10 (3203), etc. 

Figure 36 illustrates an apparatus in accordance with of an embodiment of the Dwell aspect of the 
invention having 20 detectors circumscribing an aperture (3350), each detector associated 
respectively with an indicator intersecting its associated detector, e.g. detector (3301) is associated 
15 with indicator (3303). 

Figure 37 illustrates an apparatus in accordance with of an embodiment of the Dwell aspect of the 
invention having 24 selectable regions arranged in a grid of four rows and six columns, each 
selectable region associated respectively with an indicating region intersecting its associated 
20 selectable region. For example, selectable region (3402) is associated with indicating region 
(3401). 

Figure 38 illustrates a display in accordance with of an embodiment of the Dwell aspect of the 
invention having a plurality of detectors, (2501 ), (2503), (2505), etc., arranged in a grid. Each 
25 detector is associated respectively with an indicator intersecting its associated detector., e.g. 

detector (2501) is associated with indicator (2502). Each detector is also associated respectively 
with an order entry item which may be selected by dwell. For example, detector (2501) is 
associated with a hamburger. Pointer (2512) (not drawn to scale) emits energy detectable by the 
detectors. Pointer (2512) may be housed in stationary housing (2514) (not drawn to scale). 

30 

Means for determining the difference between two data items or signals and means for totaling two 
or more data items or signals may each include a processing unit programmed to calculate this 
difference or total. Alternatively, the difference or total may be determined by electronic, 
mechanical, optical, or other suitable means. 
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The two step procedure described earlier for operating conventional menu-driven data entry and 
order entry systems incorporating pointing at intended selections may be simplified to a single step 
in accordance with the Dwell aspect of the invention. Referring now to Figure 38, the operator 
points pointer (2512) at an order entry item. The item brightens responsive to the signals falling on 

5 the associated detector, indicating to the operator which order entry item he is dwelling on and his 
dwell time on that order entry item. When his dwell time equals or exceeds the selection threshold, 
the order entry item is selected. The operation of such a system is intuitive and may be learned or 
relearned by pointing and dwelling. No surface is required, unlike a standard mouse. Additionally, 
an operator seated in the driver's seat of a vehicle is probably right-handed and may be making 

10 selections with his left hand, the hand closest to the window in a left-hand drive vehicle. Pointing 
and maintaining a pointer in a steady position requires less coordination than pointing and clicking. 

D. Path Directness 

15 The Path Directness aspect of the invention includes several aspects, hereinafter "subaspects", 
called Facilitated Dwell, Direction and Intersection, Direction, Appraisal and Drift Attenuation. 
The preferred embodiment and certain alternative embodiments of each of these subaspects will 
now be described. 

20 According to the Facilitated Dwell subaspect of the invention, the duration of the dwell period 

required for selection ("selection threshold period") of a menu option associated with a selectable 
region varies with the directness of the cursor's path to that selectable region. The preferred 
embodiment of the Facilitated Dwell subaspect of the invention will now be described in detail 
from a functional perspective using an example illustrated in Figure 39. In Figure 39 on display 

25 (2112) are defined corridors (2302), (2304), (2306), (2308), (2310), (2312) and (2314) lying 
between the previously selected selectable region (23 1 8) and, respectively, selectable regions 
(2342), (2344), (2346), (2348), (2350), (2352) and (2354). Each selectable region is associated 
respectively with a selection threshold period. Each cursor location in a cursor path, for example 
cursor path (2322), may slightly decrease one or more the selection threshold periods, except for 

30 the selection threshold period associated with the previously selected selectable region, selectable 
region (23 1 8) in this example. The effect of a cursor location depends, in the preferred 
embodiment, on whether that cursor location intersects one of the corridors. If it does, as for 
example location (2324) intersects corridor (2308), then the selection threshold period associated 
with the selectable region to which the intersected corridor leads, (2348) in this example, is 
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decreased, preferably to a limit of approximately 20% of the initial value of the selection threshold 
period so that some period of intersection of the cursor and the intended selectable region is still 
required for selection. A changed selection threshold period is preferably indicated by a change in 
the brightness of the selectable region associated with the changed selection threshold period. 
5 Thus, when the operator moves the cursor within a corridor, the selectable region associated with 
the corridor brightens, indicating both the target selectable region the system believes the cursor is 
headed toward and the changed selection threshold period. When the operator moves the cursor 
outside a corridor the cursor had previously intersected, the selectable region associated with the 
previously intersected corridor darkens in accord with the dynamic dwell aspect of the invention. 
10 indicating both that the system no longer believes the cursor is headed toward that selectable region 
and the changed selection threshold period. Reducing the selection threshold period facilitates 
selection of dwell-selectable regions without unduly increasing the likelihood of erroneous 
selections, since cursor locations within a corridor evidence the operator's intention to select the 
selectable region associated with the corridor. 

15 

Preferably, corridors are hidden from view , but they may be may be shown on the display or 
shown only at certain times or under certain conditions. Corridors may have Fixed boundaries, 
depending on which selectable region has been selected, or their boundaries may be determined 
when a starting location, for example, cursor location (2316) in Figure 39, is known. Corridor 
20 shape, size, number and position about the associated selectable region may vary, as illustrated by 
the alterative embodiments shown in Figures 40 and 41. Where corridors overlap, a cursor 
location intersecting two or more corridors may be defined to be in a cursor path toward zero, one 
or more selectable regions associated with the intersected corridors. 

25 The intersection of a cursor location and a corridor is but one means of identifying which one of a 
plurality of selectable regions is most nearly along a cursor path. A cursor path may be indicated 
by an intersection of a cursor location and a predetermined region, e.g. a corridor, by a cursor 
location and a movement related signal from which may be derived a second location, or by two or 
more successive cursor locations. As used herein, successive locations include a plurality of 

30 locations distributed in time. Successive location may be, but need not be, consecutive. Given a 
location and a movement related signal or two locations, an intention to select a particular 
selectable region may be inferred, for example, by extrapolation, and the selection thresholds 
associated with either or both the intended or unintended selectable regions modified accordingly. 
As an example, assume successive cursor locations are periodically stored in a ring buffer and the 
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magnitude of the angle between two line segments, the first between the oldest cursor location in 
the ring buffer and a predetermined point in the selectable region, and the second between the 
oldest cursor location in the ring buffer and the current cursor location, is determined. The 
selectable region associated with the smallest of these angles may be considered to be the 
5 selectable region most nearly along the cursor path indicated by the first line segment. 

An alternative means of identifying which one of a plurality of selectable region is most nearly 
along a cursor path is to determine the ratio of the number cursor locations indicating a selectable 
region to the total cursor locations in the cursor path. 

10 

Figure 42 illustrates still another alternative means for identifying which one of a plurality of 
selectable regions is most nearly along a cursor path. Figure 42 shows line (2410) from starting 
cursor location (2406) to ending cursor location (2408) just within selectable region (2404). Line 
(24 1 0) is the most direct path between these two points. In this example, the actual path traveled 

1 5 by the cursor between these two points is path (2412). The ending cursor location (2408) is known 
before the selection threshold period associated with selectable region (2404) is modified. 
Identification may be made by measuring or approximating the area within region (2414) bounded 
by line (24 1 0) and cursor path (24 1 2). The smaller the area, the more direct the cursor path. 
Alternatively, identification may be made by storing a sampling among successive cursor locations 

20 along a cursor path and, in response to the cursor intersecting a selectable region, the selectable 

region most nearly along the cursor path may be identified at one or more sampled points along the 
cursor path. The identification may be based upon a plurality of cursor locations or upon a single 
cursor location and a movement related signal. Alternatively, identification may be made by 
measuring and comparing the number of times a cursor path diverges from a predetermined path 

25 toward the intersected selectable region and/or the degree of divergence of a cursor path from a 
predetermined path toward the intersected selectable region. 

Another apparatus in accord with the Path Directness aspect of the invention is illustrated in Figure 
34. In this Figure, selectable regions (3101), (3103), (3105), (3107), (3109), (311 1), (3115), 
30 (31 17), (31 19), (3121), (3125), (3127), (3129), (3131), (3133), (3135), (3139), (3141), (3143) and 
(3145) circumscribe area (3151). Area (3 1 5 1 ) and optionally the selectable regions include 
detectors for sensing radiant energy emitted from a pointer (2202) coupled to a body member of the 
operator. A computer coupled to the detectors determines which selectable region is most nearly 
along the path indicated by the body member of the operator. Responsive to the indicated path, the 
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embodiment may facilitate the selection of one of the selectable regions by reducing a selection 
threshold, may select a selectable region upon intersection of the point indicated by the pointer 
(2202) and a selectable region, or may select a particular selectable region in advance of 
intersection of the point indicated by the pointer (2202) and the particular selectable region. 

5 

A move direction of a body member of an operator may be determined in any way that a cursor 
path may be determined, including sampling among data indicative of position of the body 
member. In determining move direction of a body member, data indicative of body member 
positions may serve the same function as cursor locations in indicating a path toward a selectable 
10 region. Position indicating means, used, for example, in indicating a position of the body member, 
includes each of the means for indicating that a selection event has occurred. 

As an alternative to decreasing a selection threshold period, an embodiment may include a plurality 
of selection thresholds, each associated respectively with a selectable region. One or more of the 
15 selection thresholds may be increased when the direction of cursor movement does not indicate a 
path toward the associated selectable region. 

Means for indicating which one of the plurality of selectable regions is most nearly along the 
cursor path includes each of the means for indicating an intersection of the location indicated by 
20 the movement related signal and a selectable region. 

According to the Direction and Intersection subaspect of the invention, the selection threshold 
period is completely satisfied in response to a cursor path to a particular selectable region, so that 
when the cursor intersects the particular selectable region, that region is selected. Preferably, the 
25 selection threshold period is completely satisfied in response to a measure of directness of a cursor 
path to a particular selectable region equalling or exceeding a predetermined measure of directness. 
In circumstances where the measured directness is less than the predetermined measure, a dwell 
period is required for selection of the particular selectable region. 

30 According to the Direction subaspect of the invention, a selectable region is selected in response to 
a cursor path to that selectable region, in advance of an intersection of the cursor and that selectable 
region. Preferably, the selectable region is selected in response to a measure of directness of a 
cursor path to a particular selectable region equalling or exceeding a predetermined measure of 
directness. In circumstances where the measured directness is less than the predetermined 
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measure, a dwell period is required for selection of the particular selectable region. 



According to the Appraisal subaspect of the invention, the directness of a cursor path to a 
selectable region is measured. Preferably, the means for measuring the directness of a cursor path 
5 includes each of the means for identifying which one of a plurality of selectable regions is most 
nearly along a cursor path. Thus, the particular means for identifying which one of a plurality of 
selectable regions is most nearly along a cursor path which best correlates with an operator's 
intended target selectable region may be identified. 

10 As earlier described, some NMD operators have relatively unimpaired directional control, despite 
having other movement disorders. The Facilitated Dwell, Direction and Intersection, and Direction 
subaspects of the Path Directness aspect of the invention utilize that capability for computer access. 
Specifically, the ability of an operator to move a cursor in a direct path toward a selectable region 
is used to facilitate selection of that selectable region. The selectable region is selected more 

15 quickly than in conventional systems utilizing selection by dwell, increasing operator productivity. 
In addition, when selectable regions are located in accordance with the Perimeter Menu aspect of 
the invention, a cursor path toward a selectable region is often unambiguous, since usually there is 
only one selectable region along a cursor path, and a large rectangular area on the display is 
available for the output of an application program and is not obstructed by the menu. In certain 

20 embodiments in accordance with the Facilitated Dwell. Direction and Intersection, and Direction 
subaspects of the Path Directness aspect of the invention, the operator may receive an indication of 
which selectable region the system believes the operator is moving the cursor toward. The 
operator may adjust the cursor path in response to this feedback and thus move the pointer more 
accurately. Additionally, in embodiments in accordance with both the Facilitated Dwell subaspect 

25 of the Path Directness aspect of the invention and the Dwell aspect of the invention, the operator 
may receive an indication of the dwell time required to select the selectable region most nearly 
along the cursor path as the required dwell time changes in response to the cursor path. 

The preferred embodiment of the Drift Attenuation subaspect of the invention will now be 
30 described in detail from a functional perspective using an example illustrated in Figure 43. Figure 
43 depicts the upper right corner of a display (2112) having two selectable regions (0434) and 
(0430) thereon. Assuming, for purposes of this example, that a movement related signal indicates 
the path shown from point (0402) to point (0404) ("first segment") at a relatively high velocity and 
from point (0404) to point (0408) ("second segment") at a relatively low velocity. The path of the 
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first segment is relatively direct, the path of the second segment, relatively meandering. During 
receipt of the movement related signal for the first segment, the cursor preferably tracks the exact 
path indicated by the movement related signal. During receipt of the movement related signal for 
the second segment, the movement of the cursor is attenuated, preferably so that the cursor does 
5 not leave selectable region (0434) until it is selected. 

Many NMD operators are unable to cleanly stop movement of a body member, resulting in a 
relatively slow or meandering path being indicated by the movement related signal, According to 
the Drift Attenuation subaspect of the invention, drift, that is. unintentional movement, indicated 

10 by the movement related signal following intentional movement is distinguished from the 

intentional movement, and cursor movement responsive to the drift is attenuated relative to cursor 
movement responsive to intentional movement. Thus unintentional movements of NMD operators 
are filtered so that the cursor is displayed closer to the location intended by the operator and 
drifting of the cursor into a nearby, but unintended, selectable region, is avoided, resulting in fewer 

15 errors due to unintended selections. 

The preferred embodiment of each subaspect of the Path Directness aspect of the invention will 
now be described in detail from an implementation perspective. Preferably, the Facilitated Dwell 
subaspect is implemented by modifications to the access program (1206) described in the detailed 

20 description of the Perimeter Menu aspect of the invention. The modifications required are: (1) 
Additional state processing should be added to STSELECTED to create a Windows® region in 
the shape of a corridor starting a predetermined distance from the selected selectable region to each 
of the other selectable regions; (2) The event data structure should be expanded to accommodate an 
event for a selectable region along a cursor path; (3) The procedure CreateEvent should be 

25 changed so that, before setting an event to EVDECAY or EV_MOVEMENT, a test is made for 
the intersection of the cursor hotspot and each corridor, and, on finding such an intersection, a new 
event, EVCORRIDOR, is stored in the event data structure indicated by pEvent for the selectable 
region associated with the intersected corridor; (4) A column should be added to the state table so 
that each state which in the prototype may receive either EV MOVEMENT or EV DECAY will 

30 on receipt of EV CORRIDOR drive that state machine to STCORRIDOR; (5) An additional 
variable, Corridorlncrement, preferably having an initial value one half the value of Increment, 
should be added the set of data associated with each state machine; and (6) A new state, 
ST CORRIDOR. should be added to the procedure PocketFsm. The pseudo-code for state 
processing in ST_CORRIDOR follows: 
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case ST_CORRIDOR: 

/* set state to previous state */ 
/* in preparation for the next */ 
/* state transition */ 
5 set pPocket->State to pPocket - >PreviousState 

increment pPocket- >Color by 
pPocket ->CorridorIncrement, but not above 
pPocket ->InitialColor plus 80% of the 
difference between pPocket - >CrestTide and 
10 pPocket- >InitialColor 

if pPocket ->Color was changed 

set pPocket- >f Paint to TRUE 
set f Internal Event to TRUE 
set Event to EV_NULL 
15 break 

An example of the selection of selectable region (2346) in accord with the preferred embodiment of 
the Facilitated Dwell subaspect of the invention will now be described with reference to Figure 39, 
assuming that the prototype has been modified as described, that the operator has just selected 

20 selectable region (23 18), that ail state machines are in state STRESET, and that the operator 
moves the cursor hotspot from a location inside selectable region (23 1 8) to location (23 16), a 
location just outside selectable region (23 1 8). On receipt of the next WM_TIMER message, 
procedure PocketFsm is called with the indicator for the state machine associated with selectable 
region (23 1 8) and with event EV CROSS OUT. Event E V_CROSS_OUT drives this state 

25 machine from its current state, ST RESET, to ST EBB TIDE. The pseudo-code for 

STEBBTIDE in procedure PocketFsm is a break statement, indicating that no state specific 
action is taken at this time, other than the transition to ST_EBB_TIDE. Control returns to 
Windows® (1204). 

30 Shortly before or shortly after the state machine associated with selectable region (23 1 8) receives 
EV_CROSS_OUT all other state machines each receive EV MOVEMENT and each makes the 
transition from ST RESET to ST EBB TIDE. 

Assume that the operator moves the cursor hotspot along path (2322) toward location (2332), a 
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location outside all corridors, and then the next WMTIMER message is received. All state 
machines receive EVMOVEMENT and are driven from ST_EBB_TIDE to STDECAY to 
ST_EBB_TIDE . The state processing associated with these states has been described in the 
description of the Perimeter Menu aspect of the invention. 

5 

Assuming that the operator moves the cursor hotspot along path (2322). the transition from 
ST_EBB_TIDE to ST DECAY to ST EBB TIDE is repeated every 54 milliseconds for each 
state machine until the operator moves the cursor hotspot beyond location (2332). At this time the 
procedure CreateEvent determines that the cursor hotspot intersects corridor (2308) and that 

10 consequently EVCORRIDOR should be sent to the state machine associated with selectable 
region (2348). EV_CORRIDOR drives this state machine from STEBBTIDE to 
ST_CORRIDOR. Stepping through the pseudo-code for ST_CORRIDOR, the ST_CORRIDOR 
state sets pPocket->State to the value stored in pPocket->PreviousState, increments 
pPocket->Color by pPocket->CorridorIncrement. but not above pPocket->lnitialColor plus 80% of 

15 the difference between pPocket->CrestTide and pPocket->lnitialColor. sets pPocket->fPaint to 

TRUE, sets flnternalEvent to TRUE, and sets Event to EV NULL. Following the break statement, 
the while flnternalEvent condition is true and another state transition occurs. The new state is 
found at aPocketFsm[ST_EBB_TIDE][EV_NULL], which equals ST EBB TIDE. This path is 
unlike an ordinary state transition because the starting state is set by STCORRJDOR. All 

20 transitions from ST CORRJDOR share this distinction. The state machine executes the code for 
the new state, ST EBB TIDE, which is simply a break statement. The procedure PocketFsm 
determines that fPaint is TRUE, invalidates the client rectangle and exits. As a result of 
invalidating the client rectangle, Windows® (1204) sends a WM_PAINT message to the access 
program ( 1 206). On receipt of WM_PAINT, the access program ( 1 204) checks the value of fPaint 

25 for each state machine, and if TRUE, sets fPaint to FALSE and redraws the visible subregion of the 
selectable region associated with that state machine and any menu option located thereon. The 
color of the redrawn visible subregion is determined by the value of the Color variable for that state 
machine. The incremented value of pPocket->Color results in a slight brightening of selectable 
region (2348) and reduces the difference between pPocket->Color and pPocket->CrestTide, 

30 corresponding to the dwell period required to select the associated selectable region. After 
redrawing, control returns to Windows® (1204). 

Shortly before or shortly after the state machine associated with selectable region (2348) receives 
EV_CORRIDOR, all other state machines each receive EV MOVEMENT and each makes the 
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This scenario is repeated at 54 millisecond intervals while the cursor hotspot travels along path 
(2322) to location (2326), a location intersecting corridor (2308). Between this location (2326) and 
5 location (2328). a location intersecting corridor (2306), along path (2322), the procedure 

CreateEvent determines that all state machines should receive EV_MOVEMENT, driving each of 
them from their current state to ST DECAY and back to their current state. As described in the 
example in the detailed description of the Perimeter Menu aspect of the invention, STDECAY 
state processing darkens the selectable region associated with the state machine, but not below a 

10 predetermined brightness represented by the variable InitialColor. Thus selectable region (2348) 
darkens when the cursor hotspot no longer intersects corridor (2308). From location (2328) to 
location (2330), the procedure CreateEvent determines that the state machine associated with 
selectable region (2346) should received EVCORRIDOR and all other state machines 
EV MOVEMENT. Consequently, selectable region (2346) gradually brightens up to a ceiling 

15 represented by pPocket->InitialColor plus 80% of the. difference between pPocket->CrestTide and 
pPocket->InitialColor. The duration of dwell time required for selection of selectable region 
(2346) is thus reduced to approximately 20% of the dwell period required without Facilitated 
Dwell. 

20 Preferably, the Direction and Intersection subaspect of the invention is implemented by making the 
changes to the prototype described for the Facilitated Dwell subaspect, except that, in incrementing 
pPocket->Color in STCORRIDOR. the upper limit for pPocket->Color in STCORRIDOR state 
processing is pPocket->CrestTide minus pPocket->lncrement. Assuming these changes, a 
selectable region whose associated Color variable is at this upper limit is selected during 

25 processing of the WM TIMER message immediately following the intersection of the cursor 
hotspot and the selectable region. 

Preferably, the Direction subaspect of the invention is implemented by making the changes to the 
prototype described for the Direction and Intersection subaspect of the invention, except that ( 1 ) 
30 the corridors are narrow, as illustrated in Figure 40; and (2) the pseudo-code for state processing in 
ST CORRIDOR is as follows: 

case ST_CORRIDOR: 

/* set state to previous state */ 



76 



PCT/US95/03591 

/* in preparation for the next */ 
/* state transition */ 
set pPocket->State to pPocket->PreviousState 
increment pPocket->Color by 
pPocket->Corridor Increment but not above 
pPocket- >Ceiling 
if pPocket->Color was changed 

set pPocket->f Paint to TRUE 
set f IntemalEvent to TRUE 

if pPocket-^Color was changed from a value 
below 

pPocket->CrestTide to a value greater than or 
equal to 

pPocket->CrestTide 

set Event to EV_STEP_UP 

else 

set Event to EV_NULL 
break 

20 State ST_CORRIDOR may now generate the internal event EV_STEP_UP, as state STDWELL 
does in the detailed description of the Perimeter Menu aspect of the invention. A state machine in 
state ST_EBB_TIDE having a Color variable equal to or exceeding the CrestTide variable, will 
transition via PocketFsm[ST_EBB_TlDE][EV_STEP_UP] to ST SELECTED. and perform the 
ST_SELECTED state processing described in the detailed description of the Perimeter Menu 

25 aspect of the invention. 

Preferably, the Appraisal subaspect of the invention is implemented by making the changes to the 
prototype described for the Facilitated Dwell subaspect, except that (1 ) the corridors are visible on 
the display (2112), (2) one of the selectable regions is designated to be the target selectable region 
30 and this is indicated to the operator, (3) cursor locations are stored in memory (2106), and (4) 
following an intersection of the cursor hotspot and the target or selection of a selectable region 
other than the target, path directness is measured in accordance with the stored cursor locations. 

The preferred embodiment of the Drift Attenuation aspect of the invention will now be described in 
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detail from an implementation perspective. Preferably, the Drift Attenuation aspect is 
implemented by modifications to the access program (1206) described in the detailed description of 
the Perimeter Menu aspect of the invention, modified as described above in the description of the 
implementation of the preferred embodiment of the Facilitated Dwell subaspect of the invention 
and further modified as follows: (1) add two booleans to the data set associated with each state 
machine, one called fDirectPath, the other fAttenuateDrift, and initialize each of them in all state 
machines to FALSE; (2) in both STDECAY and ST_CORRIDOR, set both pPocket->fDirectPath 
and pPocket->fAttenuateDrift to FALSE; (3) append to ST ENTRY state processing 
corresponding to the following pseudo-code: 

if pPocket- >Color equals or exceeds 

(pPocket->InitialColor + (0.5 + (pPocket- >CrestTide - 
pPocket- >InitialColor) ) ) 

set pPocket - >fDirectPath to TRUE 

(4) append to ST_DWELL state processing corresponding to the following pseudo-code: 

if the present cursor position intersects the edge of 

the Windows® cursor clipping rectangle and 

pPocket ->fDirectPath is TRUE 

set pPocket- >f AttentuateDri ft to TRUE 

if pPocket->f AttentuateDrift is TRUE 

store the difference in each of the x and y 
coordinates ("delta") between consecutive cursor 
locations in a circular buffer accommodating the 
last ten deltas, overwriting the oldest delta with 
the newest delta 

if ten deltas have been accumulated 

calculate the average acceleration indicated 
by the last ten deltas 

if the average acceleration is negative 

set the cursor at the location one half 
the distance between the current cursor 
location and the previous cursor location 
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display the cursor at this new cursor 
location 

E. Intersection 

5 

The preferred embodiment of the Intersection aspect of the invention will now be described in 
detail from a functional perspective using an example depicted in Figures 44, 45, 46. Figure 44 
shows the display (21 12) of a general purpose computer system (221 8 in Figure 1 5) to which is 
coupled a pointer (2202). Eight regions are delimited with respect to the display (2112) and 

10 together circumscribe region (2650) on the display (21 12). Four of these regions, (2602), (2612). 
(2622), and (2632), are entirely on the display (21 12). Each of the other four regions respectively 
includes both a visible subregion ((2606), (2616), (2626), and (2636)) on the display (2112) and an 
invisible subregion ((2604), (2614), (2624), and (2634)) adjacent and outside the display (21 12). 
Assume, for example, that the pointer is indicating location (2656) on the display (2112) and that 

15 the operator moves the pointer so that the location indicated by the pointer first intersects one of 
the regions at location (2652) in region (2612). Upon this intersection, the display changes to that 
shown in Figure 45. Figure 45 depicts a selectable region consisting of the union of invisible 
subregion (2714) and visible subregion (2716), hereinafter referred to as selectable region 
(2714/2716). Selectable region (2714/2712) is associated with menu option "sumac". In the 

20 preferred embodiment, the operator may select menu option "sumac" by dwelling on any part of the 
selectable region for the selection threshold period. Assuming, for example, that instead of moving 
the pointer from a position indicating location (2656) to location (2652), the operator instead 
moves the pointer so that the location indicated by the pointer first enters one of the regions at 
location (2654) in region (2614/2616). Upon this intersection, the display changes to that shown in 

25 Figure 46. Figure 46 depicts a selectable region consisting of the union of invisible subregion 
(2814) and visible subregion (2816), hereinafter referred to as selectable region (2814/2816). 
Selectable region (2814/2816) is associated with menu option "vort<space>x". In the preferred 
embodiment, the operator may select menu option "vort<space>x" by dwelling on any part of the 
selectable region for the selection threshold period. 

30 

In the preferred embodiment, each of the other seven regions shown in Figure 44 is associated with 
a selectable region and each selectable region is associated with a menu option. The menu options 
shown in Figure 44, in addition to "sumac" and "vort<space>x" are "wizen", "backspace", "words", 
"talk", ldhbfk" and "ypgqj,". The operator may select the menu option associated with any one of 
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the selectable regions by moving the location indicated by the pointer from circumscribed region 
(2650) into the region associated with the selectable region and then dwelling on the selectable 
region for the selection threshold period. Each selectable region preferably includes all the area of 
its associated region. 

An operator having impaired ability to maintain a body member in a steady position but who can 
control the point at which the location indicated by a body member enters a region may, in accord 
with the Intersection aspect of the invention, use his relatively unimpaired motor capability to 
selectively enlarge a selectable region or determine which of two or more selectable regions will 
occupy a predetermined area, thus making it easier for him to select. If the general purpose 
computer system of the preferred embodiment is coupled to a speech synthesizer and the menu 
options are letters or words, an operator with impaired speech may select or spell words and speak 
them. 

Figures 47 and 48 illustrate the upper right corner of a display in accordance with an alternative 
embodiment of the Intersection aspect of the invention. Figure 47 depicts the upper right corner of 
a display (2112) of a computer system (2116) having thereon two regions, (1404) and (1402), each 
associated respectively with a selectable region, each selectable region associated respectively with 
menu options "bdfhkl" and "< S pace>cmnrst". The selectable region associated with region (1402) 
includes all of region (1402). The selectable region associated with region (1404) includes all of 
region (1404) plus area (1406) between region (1404) and the right edge of the display (2112). 
Assuming the operator uses a pointer to indicate a location on the display and that the location to 
first intersect the union of regions (1402) and (1404) intersects region (1402). the display remains 
as shown in Figure 47. Dwelling in area ( 1 406) operates to select the menu option 
"<space>cmnrst". If instead the location to first intersect the union of regions (1402) and (1404) 
intersects region (1404), the display changes to that shown in Figure 48. Dwelling in area (1406) 
then operates to select the menu option "bdfhkl". 

The preferred embodiment of the Intersection aspect will now be described in detail from an 
implementation perspective. The Intersection aspect is preferably implemented by modifications to 
the access program (1206) described above in the detailed description of the Perimeter Menu 
aspect of the invention. The modifications are: (1) add a row to the state table aPocketFsm at row 
index 18 for a new state. ST_EXIT and initialize the value of aPocketFsm[ST_EXIT][EV_NULL] 
to 4. the value of ST_LOW_TIDE; (2) at initialization time, change 
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aPocketFsm[ST_LOW_TIDE][EV CROSS_OUT] to 18, the value of ST_EXIT; (3) at 
initialization time for each state machine, (a) if the visible subregion associated with the state 
machine does not abut the edge of the display, set Anterior to TRUE, otherwise set Anterior to 
FALSE; (b) initialize iAdjacentPocket to the index of the state machine associated with the 

5 adjacent region. For example, assuming the index of the state machine associated with region 
(2614/2616) in Figure 44 is 2 and the index of the state machine associated with region (2612) in 
Figure 44 is 3, iAdjacentPocket in the data set associated with the state machine associated with 
region (2614/2616) is initialized to 3 and iAdjacentPocket in the data set associated with the state 
machine associated with region (2612) is initialized to 2: (c) initialize the second array of points to 

10 define the boundaries of the region associated with the selectable region associated with each state 
machine, for example, region (2612), and initialize the first array of points to define the boundaries 
of the visible subregion of the selectable region associated with the state machine, for example, 
visible subregion (2712); (4) remove creation of Windows® regions corresponding to visible 
subregions from the access program (1206) initialization; (5) append to ST ENTRY state 

1 5 processing corresponding to the following pseudo-code: 

if pPocket->f Interior equals TRUE 

if pPocket ->hRegion is not NULL 

delete the Windows® region having the handle 
20 pPocket->hRegion 

create a Windows® region having the boundaries 
defined by the second array of points associated 
with this state machine and set pPocket- >hRegion 
to the handle to this region 
25 set pPocket- >f Paint to TRUE 

send EV_RESET to all state machines except this one 
and the state machine having the index 
pPocket- >iAdjacentPocket 

set pAdjacent Pocket to point to the data set 
30 associated with the state machine having the index 

pPocket - >iAd j acentPocket 

delete the Windows® region having the handle 

pAdj acentPocket- >hRegion 

set pAdj acentPocket- >hRegion to NULL 
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set pAdjacent Pocket ->f Paint to TRUE 

(4) append to STRESET state processing corresponding to the following pseudo-code: 

5 if pPocket- >hRegion is not NULL 

delete the Windows® region having the handle 
pPocket- >hRegion 
create a Windows® region having the boundaries defined 
by the second array of points associated with this 
10 state machine and set pPocket- >hRegi on to the handle 

to this region 

An example of the selection of selectable region in accord with the Intersection aspect of the 
invention will now be described with reference to Figures 44 and 45. Following initialization. 

15 assume that the operator moves the pointer (2202) from a position indicating location (2656) 
toward location (2652) in Figure 44. At the next expiration of the cursor polling timer, the 
procedure CreateEvent sends an EV_MOVEMENT event to the state machine associated with the 
selectable region associated with the menu option "sumac". The event EV_MOVEMENT drives 
this state machine from its current state, ST_RESET, to ST_EBB_TIDE. The pseudo-code for 

20 ST_EBB_TIDE in procedure PocketFsm is a break statement, indicating that no state specific 
action is taken at this time, other than the transition to STEBBTIDE. Control returns to 
Windows® (1204). At the expiration of the cursor polling timer after the location indicated by the 
pointer reaches location (2652), the procedure CreateEvent sends event EVDWELL to the state 
machine associated with the selectable region associated with the menu option "sumac". 

25 EV_DWELL drives this state machine from STEBBTIDE to STENTRY. Following the 
pseudo-code for ST ENTRY state processing shown above and in the Procedure PocketFsm, 
flnternalEvent is set to TRUE, Event is set to EVNULL, the Windows® region corresponding to 
region (2612), ie. the Windows® region having the handle pPocket->hRegion, is deleted and a new 
Windows® region having the boundaries indicated in the first array of points, ie. visible subregion 

30 (2712) in Figure 45, is created and the handle stored in pPocket->hRegion. pPocket->fPaint is set 
to TRUE, EVRESET is sent to state machines associated with regions other than (2612) and 
(26 1 4/26 1 6). Data in the data set associated with the adjacent region (26 1 4/26 1 6) is now modified. 
The Windows® region corresponding to region (2614/2616) is deleted. hRegion for the state 
machine associated with that region is set to NULL and fPaint associated with that state machine is 
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set to TRUE. The event EV_NULL drives the state machine associated with selectable region 
(2714/2712) to ST_LOW_TIDE. The pseudo-code for STLOWT1DE is only a break statement, 
so there is no state specific action for ST_LOW_TIDE other than entry into this state. pPocket- 
fPaint is TRUE so the client rectangle is invalidated, resulting in the redrawing of Windows® 
5 regions associated with state machines having fPaint equal to TRUE so that the display (2112) 
appears as shown in Figure 45. Control returns to Windows® (1204). 

Dwelling at any location within selectable region (2714/2712) now causes the procedure 
CreateEvent to send EV DWELL to the state machine associated with this selectable region. 
10 Selection of this selectable region proceeds as described above in the detailed description of the 
Perimeter Menu aspect of the invention. 

Assuming the operator moves the location indicated by the pointer out of selectable region 
(2714/2712) prior to selection, the procedure CreateEvent sends the event EVCROSSOUT to the 
15 state machine associate with that selectable region, driving it to STEX1T. The state processing for 
ST EXIT causes the display (2112) to change to that shown in Figure 44. 

F. Alignment 

20 The embodiment of the Alignment aspect of the invention as implemented in the prototype will 
now be described in detail from a functional and implementation perspective using an example 
depicted in Figures 49, 50, and 5 1 . Each of these Figures depicts the upper right corner of a 
display (2112) having two visible subregions on the display. In these Figures, no subregions 
outside the display (2112) are shown. Alignment is achieved in several steps and requires operator 

25 interaction with the apparatus. For purposes of this example, assume that Figure 49 depicts the 

upper right corner of the display (21 12), that an operator, fitted with a head pointer, desires to keep 
the cursor on the display directly in his line of sight and that the location indicated by the pointer is 
presently 1 5 degrees to the right of the location of the arrow cursor ( 1 802). The operator now 
dwells on selectable region (1834) for an predetermined period ("the lock threshold") which 

30 preferably is significantly greater than the selection threshold period. The apparatus responds to 

this extended dwell period by changing the display to that shown in Figure 49. The arrow cursor is 
removed from the display and the lock icon (1902) is displayed in a predetermined location of the 
intersected selectable region ("lockspot") on the display. The lock icon remains on the lockspot for 
a predetermined period ("the lock period"); it does not move responsive to the operator's head 
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movement. While the lock icon is displayed, the operator turns his head, bringing his line of sight 
into alignment with the lockspot. At the expiration of the lock period, the apparatus changes the 
display to that shown in Figure 5 1 . The lock icon is erased and the arrow cursor appears in the 
lockspot, which is where the operator is now looking. The arrow cursor moves in response to the 
5 operator's head movement. 

An operator who loses alignment between location indicated by his pointer and the cursor may thus 
initiate an alignment sequence, and then, by moving his head or other body member when the 
prototype indicates he should do so by displaying the lock icon, regain alignment. That the lock 
10 icon is displayed indicates to the operator that he can align his head or other body member. The 
position of the lock icon indicates to the operator the location on the display with which he should 
align his head or other body member. This is the location where the arrow cursor will appear at the 
expiration of the lock period. 

15 The Alignment aspect will now be described in detail from an implementation perspective as 

implemented in the prototype. The Alignment aspect of the invention is implemented an integral 
part of the state machine described in the detailed description of the Perimeter Menu aspect of the 
invention, using the same state machines and initialization, with the exception that 
aPocketFsm[ST_CREST_TIDE][EV_CEILING] is changed from 9, the value of ST_DWELL, to 

20 14, the value of ST_BEGIN_LOCK. The operator initiates the alignment process by moving the 

cursor so that the cursor hotspot intersects a selectable region. As an example. Figure 49 shows the 
arrow cursor (1802) intersecting selectable region (1834). The operator then dwells on the 
selectable region for the lock threshold, preferably at least one second greater than the selection 
threshold period. This operator action causes the state machine associated with the intersected 

25 selectable region to reach state STCRESTTIDE. On receipt of the first WM TIMER message 
following entry into state ST_CREST_TIDE, the procedure CreateEvent creates event 
EV CEILING, which drives the state machine from state ST CREST TIDE to state 
STBEGINLOCK. The state processing within state STBEGINLOCK beeps, sets the system 
cursor location to the selectable region's lockspot, sets the system cursor to null erasing the arrow 

30 cursor, displays a the lock icon (1902 in Figure 50) on the selectable region's lockspot and 

initializes the global variable iLockCursor to the number of expirations of the cursor polling timer 
corresponding to the period of time the cursor will be locked (the "lock period") configured by the 
operator, preferably two seconds, and control is returned to Windows® (1204). Following the next 
expiration of the cursor polling timer, the procedure CreateEvent may generate an event 
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EVDECAY, EV_DWELL, EV_CROSS_OUT. EV MOVEMENT or EV CEILING. Each of 
these events drives the state machine from state ST BEGIN LOCK to state ST LOCK. Within 
state ST LOCK the system cursor is moved to the selectable region's lockspot and iLockCursor is 
decremented. Then it is determined whether iLockCursor equals zero. If not, control returns to 
5 Windows® (1204) for another iteration through ST_LOCK. During the lock period, the system 
cursor is moved to the lockspot upon every expiration of the cursor polling timer, thus inhibiting 
movement of the cursor from the lockspot so that, during the lock period, the operator may move 
the location indicated by the pointer while the lock icon remains on or very close to the selectable 
region's lockspot. When, eventually. iLockCursor is decremented to zero, the global Event is set to 

10 EV_NULL and flnternalEvent is set to TRUE so that another state transition occurs immediately. 
This transition drives the state machine from state ST LOCK to state STENDLOCK. Within 
state ST_END_LOCK, the lock icon is erased, Windows® (1204) is directed to display the arrow 
cursor (2002) at the lockspot, as shown in Figure 51, the fPaint flag associated with the intersected 
selectable region is set to TRUE so that the selectable region will be drawn, restoring the 

15 background behind the erased lock icon. Assuming the cursor hotspot remains on the selectable 
region for another 54 milliseconds, CreateEvent generates EVDWELL, which drives the state 
machine to ST_DISCARD. Referring to the pseudo-code in PocketFsm. ST_DISCARD state 
processing sets State to PreviousState, returning the state machine to ST END LOCK and in 
effect, discarding the last event. Following the expiration of the cursor polling timer after the 

20 operator moves the cursor out of the selectable region it presently intersects, the procedure 

CreateEvent generates an event EV CROSS OUT which drives the state machine from state 
ST END LOCK to state ST_SELECT_AND_OUT. The access program ( 1 206) performs the 
state processing for state STSELECTANDOUT and subsequent states as previously described 
in the description of the Perimeter Menu aspect of the invention. 

25 

In the prototype, the operator may initiate alignment by depressing any character key on the 
keyboard. On receipt of a WM CHAR message from Windows® (1204) , the access program 
(1206) removes the arrow cursor from the display and displays the lock icon at a predetermined 
location on the display. In the prototype, the predetermined location is the center of the display 
30 (211 2). After a predetermined period, two seconds in the prototype, the lock cursor icon is erased 
and the arrow cursor displayed at the predetermined location. 

In the prototype, the cursor is automatically centered if the cursor hotspot does not move for two 
minutes. Lack of movement of the cursor hotspot is detected in the procedure CreateEvent, which 
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generates the event EV_IDLE_TIMEOUT for all state machines. The state processing of each state 
machine on receipt of EV IDLE TIMEOUT depends upon its current state. State machines in 
states STINITIAL and ST RESET stay in those states. State machines in all other states in which 
an external event can be received are driven to STJDLE. Referring to the pseudo-code in 
5 PocketFsm, STJDLE state processing moves the cursor to the center of the display, sets 

flnternalEvent to TRUE and sets Event to EVNULL. PocketFsm [STI DLE] [NULL] equals 3, the 
value of ST_RESET. The access program ( 1 206) performs the state processing for the ST RESET 
and subsequent states as described in the description of the Perimeter aspect of the invention. 

10 G. Length Order 

The preferred embodiment of the Length Order aspect of the invention will now be described in 
detail from a functional and implementation perspective using an example depicted in Figure 52 
and described in the detailed description of the Location Indication aspect of the invention. 

15 Assume, for purposes of this example, that the operator has previously selected the letter "s". In 

the preferred embodiment, the string "s" is passed from the access program (1206 in Figure 16) via 
the Windows® Dynamic Data Exchange ("DDE") interface to a database program (1210 in Figure 
16), preferably the FoxPro for Windows® version 2.6 program, available from Microsoft 
Corporation, Redmond, Washington, USA. The FoxPro program looks up the record having the 

20 index "s" in a database composed of 26 letters, one for each letter of the alphabet. Each record 

includes one field for a letter of the alphabet and 12 fields containing the 12 most frequently used 
English words beginning with that letter. The 12 words in the record are ordered primarily by 
length, determined by the number of letters in each word, and secondarily by alphabetic order. The 
FoxPro program returns these 12 words to the access program (1206) and these are displayed on 

25 the display (2112) as named menu options. The twelve named menu options, ordered as described 
and depicted in Figure 52, are "so", "say", "she", "said", "show", "some", "such", "state", "school", 
"social", "service", and "student". Preferably, only the root form of inflected forms of words 
which can be created through available suffixes ("root word") may be displayed as named menu 
options, so that the limited number of available menu options in combination with the apparatus' 

30 capability to add suffixes offers a large number of inflected forms. 

Words which may be named menu options in the same menu may be ordered by any suitable 
method. Preferably ordering is done by an ordering program operating on a corpus of text or 
speech including text or speech produced by individuals whose age, sex, geographic location and 
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disability are the same as or similar to that of the operator. The ordering program determines the 
frequency of use of root words in the corpus, selects the twelve most common root words, 
beginning with every possible combination of one, two and three letters, and stores them in three 
FoxPro databases for one, two and three letter word beginnings respectively, the words in each 

5 record ordered as described above. Ordering the words prior to a request minimizes the delay 
between the operator's selection of a letter or letters and the display of the named menu options. 
Preferably, the ordering program also creates a database of records for root words beginning with 
four or more letters. Each record includes the words and its frequency of use in the corpus. When 
the operator selects four or more letters consecutively, the access program (1206) requests via DDE 

1 0 that the FoxPro program (1210) look up words starting with four or more letters in the database for 
words beginning with four or more letters, select the 12 most frequently used words matching the 
selected letters, order them as described above, and return them to the access program (1206). 

An operator searching named menu options for a desired menu option may start his search in the 
15 area on the display most likely to contain the desired menu option. Upon comparison of the length 
of the desired menu option to a displayed menu option, the operator may determine whether to 
continue his search from the displayed menu option toward the front of the list or toward the rear of 
the list, or to jump to another displayed menu option in the list Further, he may make this 
determination more quickly than if the displayed menu options were sorted conventionally. The 
20 reduced menu option search time increases the operator's productivity with respect to conventional 
menu interfaces. 

H. Location Indication 

25 The preferred embodiment of the Location Indication aspect of the invention will now be described 
in detail from a functional perspective using an example depicted in Figure 52. Figure 52 shows 
the display (21 12) of a general purpose computer system (2218 in Figure 15) having thereon eight 
selectable regions, each associated respectively with a menu option. Each of the eight selectable 
regions consists of the union of a visible subregion on the display (2112) and an invisible subregion 

30 (not shown) located outside the display (2112) and adjacent the visible subregion. Selectable region 
(4301) is associated with menu option "so", selectable region (4303) with menu option "say", 
selectable region (4305) with menu option "she", selectable region (4307) with menu option 
"suffixes", selectable region (4309) with menu option "words", selectable region (431 1) with menu 
option "some", selectable region (4313) with menu option "show" and selectable region (4315) 
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with menu option "said". Also shown on the display (2112) are 12 indicating regions, each 
associated respectively with a menu option. Indicating region (4351) is associated with menu 
option "so", indicating region (4353) with menu option "say", indicating region (4355) with menu 
option "she", indicating region (4361 ) with menu option "some", indicating region (4363) with 

5 menu option "show", indicating region (4365) with menu option "said", indicating region (4371) is 
associated with menu option "such", indicating region (4373) with menu option "state", indicating 
region (4375) with menu option "school", indicating region (4381 ) with menu option "student", 
indicating region (4383) with menu option "service" and indicating region (4385) with menu option 
"social". Each indicating region is located on the display (2112) in the region (4350) 

10 circumscribed by the selectable regions. In accord with the Location Indication aspect of the 

invention, the location of each of the indicating regions (4351), (4352), (4355), (4361), (4363) and 
(4365) indicates the location of each of the selectable regions associated with the menu option 
associated with the indicating region. These selectable regions, respectively, are (4301), (4303), 
(4305). (4311). (4313) and (4315). 

15 

In the preferred embodiment, selection of menu option "words" causes selectable region (4301) to 
be associated with menu option "such" instead of menu option "so", selectable region (4303) to be 
associated with menu option "state" instead of menu option "say", selectable region (4305) to be 
associated with menu option "school" instead of menu option "she", selectable region (4311) to be 

20 associated with menu option "student" instead of menu option "some", selectable region (4313) to 
be associated with menu option "service" instead of menu option "show", and selectable region 
(43 15) to be associated with menu option "social" instead of menu option "said". In the preferred 
embodiment, following the selection of menu option "words", the menu option newly associated 
with each selectable region is displayed on that selectable region (not shown). Following the 

25 selection of menu option "words", indicating regions (4371), (4373), (4375), (4381), (4383) and 
(4385) each indicate the location of each of the selectable regions associated with the menu option 
associated with the indicating region. 

The indicating regions and the menu options displayed thereon in Figure 52 are disproportionately 
30 large relative to the rest of Figure 52. They are approximately 1 .5 times their proportionate size. 
They are represented as shown in Figure 52 in compliance with Patent Cooperation Treaty Rules 
requiring a minimum size for letters in figures. 

In the preferred embodiment of the Perimeter Menu aspect of the invention, all the menu options 
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displayed on the selectable regions cannot be seen in a glance by many operators. However, when 
the displayed menu options are displayed on the indicating regions, for example, as shown in 
Figure 52, the displayed menu options can be seen in a glance, facilitating searching of the menu 
by the operator for his intended menu option. Since the location of each selectable region is 
5 indicated by the location of the associated indicating region, the operator may point to the intended 
selectable region without searching it out or pausing to verify that the menu option associated with 
a selectable region is the menu option he desires. The frequent operator of such a menu interface 
may habituate the process of selecting an intended menu option so he can focus his attention on 
another task while selecting the option, for example, planning his next interaction with the menu 
10 interface. 

Figure 53 depicts a display and structures in accord with an another embodiment of the Location 
Indication aspect of the invention. Selectable regions (4604), (4608), (4612), (4616), (4620), 
(4624), (4628) and (4632) are located adjacent the display (2112) and associated respectively with 

15 menu options "so", "say", "she", "suffixes", "said", "show", "some" and "words". Each selectable 
region is unbounded on the side furthest from and parallel to the edge of the display. Indicating 
regions (4351), (4353), (4355), (4357), (4367), (4361), (4363) and (4365) are associated 
respectively with menu options "so", "say", "she", "suffixes", "said", "show", "some" and "words". 
In accord with the Location Indication aspect of the invention, the location of an indicating region 

20 indicates the location of the selectable region associated with the menu option associated with the 
indicating region. 

The large size of the selectable regions outside the display, for example, as shown in Figure 53, 
facilitate selection by individuals with impaired fine motor control while the indicating regions 
25 indicate the location of each associated selectable region. 

Yet another embodiment in accord with the Location Indication aspect of the invention is shown in 
Figures 54 and 55. Figure 54 depicts a display (2112) having thereon eight selectable regions, each 
associated respectively with a menu option. As shown in the Figure, selectable region (391 1) is 
30 associated with menu option "aeiou", selectable region (3910) with menu option "gqjyvwxz", 
selectable region (3908) with menu option "setup", selectable region (3907) with menu option 
undo, selectable region (3905) with menu option "control", selectable region (3904) with menu 
option "words", selectable region (3902) with menu option "nsrm<space>tcp" and selectable region 
(3901 ) with menu option "bdfhkl". Menu option "aeiou" is associated with a submenu which 
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includes submenu options "a", "e". "i", "o" and "u". displayed on indicating regions (3930), (3928), 
(3926). (3924) and (3922) respectively. The background pattern of indicating region (3930) 
matches the background pattern of selectable region (3908), indicating that submenu option "a" 
will be associated with selectable region (3908) following selection of menu option "aeiou". 

5 Similarly, the background pattern of indicating region (3928) matches the background pattern of 
selectable region (3910), indicating that submenu option "e" will be associated with selectable 
region (3910) following selection of menu option "aeiou". Alternatively, the indication may be 
made by the size, shape, hue. brightness, contrast, dithering, fill, blinking, hatching or pattern of the 
indicating region or any part thereof, including either of the foreground and background of the 

10 indicating region. 

When the operator selects selectable region (391 1), the display (21 12) changes to that shown in 
Figure 55. Now, selectable region (3910) is associated with submenu option "e". selectable region 
(3908) with submenu option "a", selectable region (3904) with submenu option "u". selectable 
15 region (3902) with submenu option "o" and selectable region (3901) with submenu option "i". In 
accord with the Location Indication aspect of the invention, the appearance of an indicating region 
indicates the location of the selectable regions associated with the submenu option associated with 
the indicating region. 

20 The embodiment shown in Figures 54 and 55 illustrates how the Location Indication aspect of the 
invention speeds selection of a submenu option of a menu hierarchy. The operator, by observing 
the background pattern of the submenu option within the displayed menu option, may determine 
which selectable region he should next dwell on. The operator may make this determination prior 
to selection of the menu option and need not wait for the submenu options to be displayed on their 

25 associated selectable regions. In accordance with the Location Indication aspect of the invention, 
the operator may select or spell out words more quickly than with conventional automated menu 
hierarchy. Assuming the display (2112) is part of computer system, these words may be input to an 
application program, and , if the computer system is coupled to a speech synthesizer, these words 
may be spoken. 

30 

Figure 56 depicts another apparatus in accord with the Location Indication aspect of the invention. 
An example of the operation of this apparatus will now be described. Figure 56 depicts display 
area (4770) adjacent to which are located selectable regions (4701), (4703), (4705), (4707), (4709). 
(471 1), (4713) and (4715). Not shown in the Figure are three sets of eight menu options. At 
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different times during the operation of the apparatus shown in Figure 56, for example at one second 
intervals, the plurality of selectable regions is associated with a different set of eight menu options 
such that, for a one second period, each selectable region is associated respectively with one menu 
option of the associated set of menu options. On the display area are 24 indicators, each associated 
5 respectively with one of the menu options. Each indicator indicates when a selectable region is 
associated with the menu option associated with the indicator. Thus, each of the indicators (4721 ), 
(4723), (4725), (4727), (4729), (4731), (4733) and (4735) is associated respectively with a menu 
option which may be in turn associated respectively, for example, during a first one second period, 
with one of the selectable regions. The operator may select a desired one of the 24 menu options 
10 by selecting the associated selectable region during the period when the desired menu option is 
associated with the associated selectable region. 

Alternatively, the apparatus shown in Figure 56 may require two successive selection events, the 
first selection event to select a set of eight menu options or to select a set of three menu options, the 
15 set being associated with one selectable region, and the second selection event to select one of the 
selected set. For example, the first selection event may select the set of menu options associated 
with indicators (4773), (4753) and (4733), and the second selection event may select one menu 
option from this set. 

20 Still another apparatus in accord with the Location Indication aspect of the invention is shown in 
Figure 57. Figure 57 depicts display area (4770) outside of which are located selectable regions 
(5001), (5003), (5005), (5007), (5009), (501 1), (5013), (5015), (5021), (5023), (5025), (5027), 
(5029), (5031), (5033) and (5035). The location of each selectable region is indicated by an 
indicating region on the display area, (5041 ), (5043), (5045), (5047), (5049), (505 1 ), (5053), 

25 (5055), (5061), (5063), (5065), (5067), (5069), (5071), (5073) and (5075), respectively. 

The Location Indication aspect of the invention will now be described from an implementation 
perspective with reference to Figure 52. Preferably, the Location Indication aspect is implemented 
by modifications to the access program (1206) described in the detailed description of the 
30 Perimeter Menu aspect of the invention. The modifications required for the Location Indication 
aspect of the invention are: (1 ) at initialization time: (a) create twelve child edit windows, each 
corresponding to one of the indicating regions shown in Figure 52, each of the class "edit", each 
having the style (WS_CHILD | WSBORDER | WS D1SABLED | ES_MULTILINE | 
AlignmentStyle) where "|" represents a logical OR operation and where AlignmentStyle equals 



91 



WO 96/30822 PCT/US95/03S91 

ESRIGHT for child windows having right justified text and ES_LEFT for child windows having 
left justified text, as shown in Figure 52; (b) move the child edit windows so they are located in or 
near the center of the display; (c) display the child edit windows; and (d) initialize an array which 
maps each child edit window, for example, by an index between 0 and 1 1 inclusive, to an element 

5 of the aLabel array; and (2) modify the code implementing STSELECTED state processing, so 
that after changing the menu options associated with various selectable regions (by changing the 
pLabel element of the data set associated with each of the state machines to point to the menu 
option in the aLabel array to be associated with that state machine), the array mapping each child 
window to an element of the aLabel array is used to access the text of the menu option and the text 

10 of each child edit window is set accordingly. In the preferred embodiment, the six menu options 
displayed in indicating regions (4371), (4374), (4375), (4381), (4383) and (4385) are present in the 
aLabel array when Figure 52 is shown, but are not associated with selectable regions until the 
operator selects menu option "words". 

15 I. Sound Match 

The preferred embodiment of the Sound Match aspect of the invention will now be described in 
detail from a functional perspective using an example depicted in Figure 58. Figure 58 shows a 
display (2112) of a computer system (2116) having thereon six regions or sound indicators, (5801), 

20 (5803), (5805), (5807), (5809), and (581 1), each associated with a menu option, "ypgqj,", "ldhbfk", 
"wizen", "words", "vort x". and "sumac", respectively. Each menu option is displayed on its 
associated sound indicator. Each sound indicator has a distinct hue. For example, sound indicator 
(5801) may be green, (5803) white, (5805) blue, (5807) red, (5809) orange and (581 1) grey. Each 
of these sound indicators indicates a sound the operator is able to consistently produce, for 

25 example, the vowel sound e as it sounds in green, i as it sounds in white, u as it sounds in blue, e as 
it sounds in red, o as it sounds in orange and e it sounds in grey. Assuming the operator says o as it 
sounds in orange, "vort x", the menu option associated with the orange sound indicator (5809) is 
selected and the display is changed to that shown in Figure 59. In the preferred embodiment, 
submenu options space, "x", "r", "t", "v" and "o" are now associated with each of the sound 

30 indicators (5901), (5903), (5905), (5907), (5909), and (591 1) respectively. Assuming the operator 
says e as in green, submenu option space is selected and a space is input to the application program 
(6107) whose output appears in region (5850). 

Figure 60 illustrates a display and structures in accordance with an alternative embodiment of the 
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Sound Match aspect of the invention. Figure 60 depicts a display (21 12) of a computer system 
(2116). Adjacent the display (21 12) is a static display (6252) having thereon eight sound indicators 
(6204), (6208), (6212), (6216), (6220), (6224), (6228). and (6232). Each of the eight sound 
indicators is a symbol in a phonetic character set graphically representing a sound. Alternatively, 
5 the sound indicators may include a picture of an object, for example, a tree, a house, a boy, or a 
map of a country, or may include a shape, for example, a square, a circle, or a triangle, or may 
include a hue indicator, a pitch indicator, a volume indicator, a sound duration indicator, a change 
in pitch indicator, or a change in volume indicator. Eight menu options. "vort<space>x", "sumac", 
"wizen"', undo, "words", "talk", "ldhbfk" and "ypgqj,". are displayed on the display (21 12). each 
10 adjacent and associated respectively with one of the sound indicators, the eight menu options 

together circumscribing region (6250) on the display. The operator may select any one of the menu 
options by producing the sound indicated by the sound indicator associated with the menu option. 
In response to a menu selection, submenu options of the selected menu option may be displayed 
and associated respectively with a sound indicator. 

15 

An operator with impaired speech but who is able to consistently produce a relatively small number 
of sounds distinguishable by conventional speech recognition means may, in accord with the Sound 
Match aspect of the invention, select from among the small number of menu options by using the 
sounds he can produce. Assuming the display (2112) is part of computer system, the menu option 
20 may represent inputs to an application program, and, if the computer system is coupled to a speech 
synthesizer, the menu option may represent words to be spoken. 

Figure 61 illustrates a display and structures in accordance with an alternative embodiment of the 
Sound Match aspect of the invention. Figure 61 depicts a display (21 12) of a computer system 

25 (2116) having thereon six square sound indicators, (6401), (6403), (6405), (641 1), (6413), and 
(6415) arranged in two columns of three sound indicators. Each sound indicator abuts a sound 
indicator in the other column, the sound indicator located above it, if any, and the sound indicator 
located below it, if any. Assume that the operator is entering Chinese, that he uses a keyboard to 
enter a phonetic unit and intonation according to the Pin Yin coding method for the Chinese 

30 language, and that he enters the distinct sound "fid", each of the sound indicators shown in Figure 
61 is associated respectively with a menu option, each of the plurality of menu options having a 
common characteristic. In this example, the menu options are homophones and the common 
characteristic is a phonetic unit and intonation but may be a phonetic unit alone, an intonation 
alone, a stroke used to draw an ideograph, a number of horizontal strokes, a number of vertical 
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strokes, a number of total strokes, a stroke order, a radical, a part of speech, an ideograph, a kana, a 
diacritic, a classification of a part of an ideograph or other characteristic of a class of ideographs. 
Each of the sound indicators has a distinct hue. The operator may select any one of the displayed 
menu options, according to the Sound Match aspect of the invention, by speaking the sound 
5 associated with the sound indicator associated with the desired menu option. For example, assume 
the operator says, "blue" or a translation thereof, preferably a Chinese translation in this example. 
The ideograph located on the blue sound indicator is selected. In this example, this ideograph is 
input to a word processing program and appears on the display (21 12). 

10 The Sound Match aspect of the invention thus allows an operator to select from an option from a 
menu, using speech recognition means, whether or not the menu options are homophones. The 
operator does not need to use his hands to make this selection and so may keep both his hands on 
the home row of the keyboard, in preparation for entering the next common characteristic, or, if 
specifying the common characteristic by voice, may select a menu option without interrupting the 

1 5 manual activity he's engaged in. 

The preferred embodiment of the Sound Match aspect of the invention will now be described in 
detail from an implementation perspective, beginning with the hardware and software operating 
environment which will now be described with reference to Figure 62 which depicts a speech 

20 recognition system (6001) including a computer system (2116) and keyboard (2210), as earlier 
described, a sound board (6007) and a microphone (6009). The conventional computer system 
(2116) preferably includes an 80486 CPU running at 33MHz or faster, and is provided with Dragon 
Dictate Power Edition software, available from Dragon Systems, Inc., Newton, Massachusetts, 
USA. Preferably, the sound board is the Audio Capture Playback Adaptor and the microphone is 

25 the Shur Headset microphone, both available from Dragon Systems, Inc. Figure 63 depicts the 
software environment of the preferred embodiment, which includes the earlier described 
Windows® version 3.1 operating system (1204), an optional Windows® application program 
(6107), the Windows® Dragon software driver (6101 ) included in the Dragon Dictate Power 
Edition software, the Dragon Application (6103) included in the Dragon Dictate Power Edition 

30 software, and a speech recognition access program (6105). The Dragon Application (6103) is 
configured to match a sample of each of six sounds distinguishable by the Dragon Application 
(6103) which the operator can consistently produce. Using the example described above, these are 
the vowel sounds produced by the operator of e as it sounds in green, i as it sounds in white, u as it 
sounds in blue, e as it sounds in red, o as it sounds in orange and e it sounds in grey. Each of these 
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sounds is associated respectively with an identifier, for example, a number or a sequence of one or 
more characters. The speech recognition access program (6105) is preferably a Windows® 
application program developed using, in part, the Voice Tools Software Development Kit available 
from Dragon Systems, Inc. At initialization time, the speech recognition access program (6105) 
5 defines an array of data structures defining the menu and submenu options and the menu hierarchy. 
For example, one of the elements of this array determines that, on selection, certain actions are to 
be taken, for example, inputting text to an application program, and that certain submenu options 
and related data are to be associated with certain child edit windows. Also at initialization time, 
the speech recognition access program (6105) initializes callback procedures using the Dragon 

10 Application's Application Program Interface to receive notification from the Dragon Application 
when a sound has been matched. Also at initialization time, the main window of the speech 
recognition access program (6105) is preferably sized to just encompass the sound indicators 
shown in Figure 58. Also at initialization time, the speech recognition access program (6105) 
creates six child edit controls, each corresponding to one of the sound indicators shown in Figure 

15 58. Each of the child edit controls has the background color described above and a text color of 
black or white depending upon which provides better contrast against the background color of the 
child edit control, and each is located on the display (2112) as shown in Figure 58. Also at 
initialization time, the menu options of the initial menu are each associated respectively with one of 
the child edit controls. After initialization, the speech recognition access program (6105), upon 

20 notification from the Dragon Application that a sound has been received and a match attempted, 

sequentially searches the list of identifiers matched to the sound by the Dragon Application (6103), 
starting with the best match, until it finds an identifier corresponding to any one of the six sounds 
distinguishable by the Dragon Application (6103). Assuming the operator says o as it sounds in 
orange, the Dragon Application (6103) provides to the speech recognition access program (6105) a 

25 list of matches including, before any other identifier corresponding to any one of the six sounds 

distinguishable by the Dragon Application (6103), the identifier associated with child edit control 
(5809). This child edit control is currently associated with menu option "vort x", the matched 
menu option. The speech recognition access program (6105) then sets the text of each of the child 
edit controls to one of the submenu options associated with the matched menu option. In the 

30 example above, submenu options space, "x", "r", "t", "v" and "o" are associated with each of the 

child edit controls respectively and the text of the associated child edit control is set to the submenu 
option. Assuming the operator says e as in green, submenu option space is the matched submenu 
option and a space is input to the application program (6107) whose output appears in region 
(5850). Preferably the application program (6107) is an application program capable of executing 
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a WM_SIZE command so that the speech recognition access program (6105) may size the windows 
of the application program (6107) to fit neatly in region (5850) and is capable of executing 
WM CHAR messages so that the speech recognition access program (6105) may input characters 
to the application program (6107). 

5 

J. Ideographic Language 

The preferred embodiment of the Ideographic Language aspect of the invention will now be 
described in detail from a functional perspective using an example depicted in Figure 64. Figure 

10 64 shows the display (2112) of a general purpose computer system (2218 in Figure 15) and 12 

selectable regions. Each of the 12 selectable regions consists of the union of a visible subregion on 
the display (2112) and an invisible subregion located outside the display (2112) and adjacent the 
visible subregion. For example, the selectable region in the upper left corner both above and below 
the top of the display (2112) in Figure 64 consists of invisible subregion (3604) and visible 

15 subregion (3606), and hereinafter is referred to as selectable region (3604/3606). The other 
selectable regions shown in Figure 64, proceeding counter clockwise from selectable region 
(3604/3606) are (3608/3610), (3612/3614), (3616/3618), (3620/3622), (3624/3626), (3644/3646), 
(3648/3650), (3652/3654), (3656/3658), (3660/3662) and (3664/3666). In Figure 64 each of the 
visible subregions is adjacent an edge of the display (21 12). The selectable regions together 

20 circumscribe region (3680) in the center of the display. Also shown on the display (2112) within 
region (3680) in Figure 64 are ten square indicating regions arranged in two columns of five 
indicating regions. Each indicating region abuts an indicating region in the other column, the 
indicating region located above it, if any, and the indicating region located below it, if any. In the 
preferred embodiment, each indicating region indicates by its location the location of a respectively 

25 associated selectable region, in accord with the Location Indication aspect of the invention. For 
example, the uppermost indicating region in the left column of indicating regions (3605) is 
associated with the uppermost selectable region (3604/3606) on the left side of the display. 
Indicating region (3609) located immediately below indicating region (3605) is associated with 
selectable region (3608/3610) located immediately below selectable region (3604/3606). 

30 

The operation of the example of the preferred embodiment of the Ideographic Language aspect of 
the invention will now be described. Assuming that the operator is fitted with a head pointer 
coupled to the general purpose computer system (2218). that he uses a keyboard to enter a phonetic 
unit and intonation according to the Pin Yin coding method for the Chinese language, and that he 
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enters the distinct sound "ft", each of the plurality of the selectable regions shown in Figure 64 is 
associated with one of a plurality of menu options, each of the plurality of menu options having a 
common characteristic. In this example, the common characteristic is a phonetic unit and 
intonation but may be a phonetic unit alone, an intonation alone, a stroke used to draw an 

5 ideograph, a number of horizontal strokes, a number of vertical strokes, a number of total strokes, a 
stroke order, a radical, a part of speech, an ideograph, a kana, a diacritic, a classification of a part of 
an ideograph or other characteristic of a class of ideographs The ten menu options shown in 
Figure 64 are Chinese ideographs each starting with the distinct sound "ft". Alternatively, the 
menu options may be sequences of graphic symbols including one or more kanji. In Figure 64 the 

10 Chinese ideographs are each displayed on the visible subregion of the associated selectable region 
and on the indicating region associated with the selectable region. The remaining two of the 12 
selectable regions, (3624/3626) and (3644/3646), are associated with menu options for undo and 
for displaying more menu options, respectively. In the preferred embodiment, in response to the 
selection of the menu option for displaying more menu options, each of the selectable regions 

1 5 associated with a menu option starting with the distinct sound "ft" is associated with a menu option 
not previously displayed and the newly associated menu option replaces the old menu option on the 
display. 

Resuming now with the description of the example of the preferred embodiment, a cross hair 
20 cursor (3686) is displayed in the circumscribed region (3680) Assuming the operator desires to 
select the menu option associated with selectable region (3608/3610), he turns his head to the left 
and the cross hair cursor (3686) moves to the left, responsive to the head movement, until the cross 
hair cursor hotspot intersects selectable region (3608/3610) and he maintains the location of the 
cross hair cursor hotspot on that selectable region for the selection threshold period. The menu 
25 option associated with selectable region (3608/3610) is selected and added to text (3684) displayed 
in the circumscribed region (3680), the general purpose computer system (2218) emits an audible 
beep indicating that selection has occurred and the displayed menu options, both on the indicating 
regions and the visible subregions, are removed from the display. 

30 In the preferred embodiment, selection is made in accord with the Facilitated Dwell subaspect of 
the Dwell aspect of the invention, described above. Thus, the operator receives an indication of the 
progress of his selection by a change in appearance of the indicating region associated with the 
intersected selectable region.. Alternatively, selection may be by intersection of a location 
indicated by the at least part of a cursor and a selectable region alone, by such an intersection 
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accompanied by a switch operation, for example, a depression of a space bar on the keyboard, or by 
other suitable means. 



In accord with the Location Indication aspect of the invention, the operator sees the entire menu in 
the compact indicating regions and may discover the location of the selectable region associated 
with each menu option without having to visually scan all the visible subregions. In the preferred 
embodiment, the plurality of indicating regions may be moved to a different location on the display 
to avoid obstructing the area of the display showing most recently added graphic symbols or the 
area of the display where graphic symbols will soon be added. In accord w ith the Ideographic 
Language aspect of the invention, an operator may select from among many sequences of one or 
more ideographs without lifting either hand from the keyboard, thus speeding entry of single 
ideographs or sequences of ideographs in word processing systems for the Chinese, Japanese and 
Korean languages. Since, in the preferred embodiment, the selectable regions are located adjacent 
the edge of the display, a large rectangular region remains available on the display for the output of 
an application program. Further, if the selectable regions are located entirely outside the display, 
the indicating regions obstruct only a relatively small portion of the circumscribed region (3680), 
permitting the display of a sequences of ideographs for selection simultaneous with the display of 
previously selected ideographs, neither obstructing the operator's view of the other. If the general 
purpose computer system is coupled to a speech synthesizer and the ideographs are symbols of a 
symbol set, for example, the Blissymbolics Symbol Set, an illiterate operator, for example, a school 
child having impaired speech, may select symbols associated with words and those words may be 
spoken via the speech synthesizer. 

Figure 65 illustrates a display and structures in accordance with an alternative embodiment of the 
Ideographic Language aspect of the invention. In this Figure, 18 selectable regions (3704/3706), 
(3708/3710), (3712/3714), (3716/3718), (3720/3722), (3724/3726), (3728/3730), (3732/3734), 
(3736/3738), (3740/3742), (3744/3746), (3748/3750),.(3752/3754), (3756/3758), (3760/3762), 
(3764/3766), (3768/3770), (3772/3774) and (3776/3778) circumscribe region (3780) on the display 
(21 12). The visible subregions of four of the 18 selectable regions abut each of the top, left and 
right edges of the display (21 12). The visible subregions of six of the 18 selectable regions abut 
the bottom edge of the display (21 12). The 18 indicating regions are located within the top half of 
region (3780). 12 of the 18 indicating regions are arranged in two columns of six indicating 
regions each. The column on the left is close to the visible subregions abutting the left edge of the 
display. The column on the right is close to the visible subregions abutting the right edge of the 
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display. Two of the 18 indicating regions are located between the top indicating regions in each of 
the left and right columns. The remaining four of the 1 8 indicating regions are located between 
the bottom indicating regions in each of the left and right columns. Each of the four indicating 
regions in the top row of indicating regions indicates the location of a respectively associated 
5 selectable region along the top edge of the display. Each of the six indicating regions in the bottom 
row of indicating regions indicates the location of a respectively associated selectable region along 
the bottom edge of the display. Each of the middle four indicating regions in each of the left and 
right columns of indicating regions indicates the location of a respectively associated selectable 
region along the left and right edge of the display. The 1 6 menu options shown in Figure 65 are 
10 Chinese ideographs each of which includes the radical kou, a radical having the shape of a 

rectangle. Each ideograph is displayed on the visible subregion of the associated selectable region 
and on the indicating region associated with the selectable region. The remaining two of the 1 8 
selectable regions, (3724/3726) and (3744/3746), are associated with menu options for undo and 
for displaying more menu options, respectively. 

15 

The indicating regions (3782) in Figure 65 are disproportionately large relative to the rest of Figure 
65. They are approximately 1 .5 times their proportionate size. They are represented as shown in 
Figure 65 in compliance with Patent Cooperation Treaty Rules requiring a minimum size for letters 
in figures. 

20 

Figure 66 illustrates a display and structures in accordance with an alternative embodiment of the 
Ideographic Language aspect of the invention. Figure 66 depicts a display (21 12) of a computer 
system (2116) having thereon ten square indicating regions. (3801). (3803), (3805), (3807), (3809), 
(381 1), (3813), (3815), (3817), and (3819), arranged in two columns of five indicating regions. 

25 Each indicating region abuts an indicating region in the other column, the indicating region located 
above it, if any, and the indicating region located below it, if any. Each indicating region indicates 
the location of a respectively associated selectable region (not shown) outside the display. Assume 
that the operator is entering Chinese and uses a keyboard coupled to the computer system (2116) to 
enter the distinct sound "fu" . On each of the indicating regions is then displayed an ideograph 

30 having with the distinct sound "fu" . The operator, using a pointer coupled to the computer system 
(2116), points to the selectable region associated withthe indicating region on which is displayed 
the desired ideograph. The selectable region is selected in response to a selection event, and, in 
this example, the associated menu option is input to a word processing program and appears on the 
display (21 12). 
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Figure 67 illustrates a display and structures in accordance with an alternative embodiment of the 
Ideographic Language aspect of the invention. Figure 67 depicts a display (2112) of a computer 
system (2116) having thereon eight selectable regions (6701), (6703), (6705), (6707), (6709), 
(671 1), (6713), and (6715), each located on the display (21 12) adjacent the edge of the display 
5 (2112) and associated respectively with a menu option. Together the eight selectable region 

circumscribe region (6750) on the display. In Figure 67 each of eight menu options is displayed on 
its associated selectable region. Six of the eight menu options, associated with selectable regions 
(6703), (6705), (6707), (6701), (6715). and (6713)each represent a sequence of graphic symbols in 
the Japanese language. These sequences, in Romanized Japanese are respectively "hoka", "hoka 

10 ni", "hoka no", "hoka kara", "nanika hoka nomono", and "hoka demo nai ga". Each sequence 

includes the kanji "hoka" shown alone on selectable region (6703). The sequences are ordered by 
length, the shorter sequences on selectable regions on the left side of the display (2112) ordered by 
length from the top to the bottom of the display (21 12). the longer sequences along the right side of 
the display (2112) ordered by length from the top to the bottom of the display (21 12). The 

1 5 remaining two of the eight selectable regions, (6709) and (67 1 1 ), are associated w ith menu options 
for displaying the previous and the next display of menu options, respectively. The menu option 
associated with a selectable region may be selected by a selection event. 

The preferred embodiment of the Ideographic Language aspect of the invention will now be 
20 described in detail from an implementation perspective. Preferably, the Ideographic Language 

aspect of the invention is implemented by modifications to the access program (1206) described in 
the detailed description of the Perimeter Menu aspect of the invention, modified as described in the 
descriptions of the Facilitated Dwell subaspect and the Location Indication aspects of the 
invention. The modifications required for the Ideographic Language aspect of the invention are: (1 ) 
25 install on the general purpose computer system a font for the ideographic language of the 

embodiment; (2) at initialization time: (a) set the text of all labels for display on the selectable 
regions to null; (b) do not enable the cursor polling timer; and (c) hide the Windows® ( 1 204) 
system cursor; (3) include in the main window processing procedure of the access program (1206) 
code to process WM CHAR messages and, when a sequence of one or more WMCHAR 
30 messages indicates a common characteristic: (a) lookup sequences or representations of sequences 
of one or more ideographs (hereinafter "sequences") having the common characteristic; (b) copy 
ten of the sequences to the labels; (c) set the Windows® (1204) system cursor to the cross hair 
cursor, set the cursor location to a predetermined location near an indicating region and show the 
Windows® (1204) system cursor; (d) set fPaint to TRUE for every state machine; (e) send 
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EVRESET to every state machine; (0 display each sequence at the appropriate location in the 
indicating region; and (g) enable the cursor polling timer; and (4) add state processing to 
STSELECTED to: (a) set the text of all labels for display to null; (b) hide the Windows® (1204) 
system cursor; (c) erase all sequences from the indicating region; (d) disable the cursor polling 
5 timer; (e) set fPaint to TRUE for every state machine; (f) send EV RESET to every state machine; 
and (g) insert the selected sequence into the work space. 

In the above descriptions, there is shown and described only the preferred and certain alternate 
embodiments of each aspect of the invention, but. as aforementioned, it is to be understood that 
10 each aspect of the invention is capable of use in various other combinations and environments and 
is capable of changes or modifications within the scope of the inventive concepts as expressed 
herein. 
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Appendix I 
Procedure : PocketFsm 



5 y*****************************************************************/ 

/* */ 

/* Procedure: PocketFsm */ 

/* */ 

/* Description: State Machine for processing events to */ 

10 /* selectable regions. */ 

/* */ 

/* Input Parameters: */ 

/* pPocket pointer to selectable region data */ 

/* set */ 

15 /* Event event to process */ 

/* */ 
/**+**************************************************************/ 

PocketFsm (pPocket, Event) 

{ 

20 BOOL f InternalEvent /* declaration of local 

boolean variable */ 



set f InternalEvent to TRUE 



25 while ( f InternalEvent is TRUE) 

{ 

set f InternalEvent to FALSE 

set pPocket - >PreviousState to pPocket - >State 

set pPocket->State to aPocketFsm [pPocket - >State] [Event] 

30 

switch (pPocket->State) 

{ 

case ST_ILLEGAL_STATE: 
case ST ERROR: 
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turn off all timers 
log the error 

display an error message to the operator 
break 

case ST_INITIAL: 
break 



case ST_RESET: 

10 set time of selection to the current time 

if pPocket->Color is not equal to 
pPocket->InitialColor 

set pPocket- >Color to pPocket- >InitialColor 
set pPocket- >f Paint to TRUE 
15 set pPocket- >f Invert to FALSE 

break 



case ST_LOW_TIDE : 
break 

20 

case ST_CREST_TIDE : 
break 



case ST_BEGIN_LOCK: 
25 emit an audible beep 

set the system cursor location to the visible 

subregion ' s lockspot 

set the system cursor to null 

display the lock icon on the visible subregion 's 
30 lockspot 

/* initialize counter 
for locking */ 

set iLockCursor to cLockCursor 

break 
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case ST_LOCK: 

set the system cursor location to the visible 
subregion's lockspot 

/* on lock, counter 
5 expiration, */ 

/* transfer to the next 
state */ 
decrement iLockCursor by 1 
if iLockCursor equals 0 
10 set Event to EV_NULL 

set f InternalEvent to TRUE 
break 



case ST_END_LOCK : 
15 erase the lock icon with white 

set the system cursor to the arrow cursor 
display the system cursor on the visible 
subregion' lockspot 
emit an audible beep 
20 /* paint selectable region 

so that */ 

/* the white erasure of the 
lock */ 

/* cursor icon is 
25 overwritten */ 

set pPocket->f Paint to TRUE 
break 



case ST_SELECTED: 
30 emit an audible beep 

take the action appropriate upon selection of this 
selectable region, for example, sending data to an 
application program, sending a control sequence to 
a device coupled to the computer, or displaying 
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the selection 

set pPocket- >f Invert to TRUE 

if appropriate, change the menu options associated 
with various selectable regions and set fPaint to 
5 TRUE for the state machines associated with those 

selectable regions 
set Event to EV_NULL 
set f InternalEvent to TRUE 
break. 

10 

case ST_SELECT_AND_OUT : 

send EV_RESET then EV_MOVEMENT to all state 
machines, resetting them and indicating that the 
operator is moving the cursor 
15 break 



case ST_DECAY: 

/* set state to previous 
state */ 

20 /* in preparation for the 

next */ 

/* state transition */ 
set pPocket ->State to pPocket- >PreviousState 
decrement pPocket- >Color by pPocket - >Decrement , 
25 but not below pPocket - >InitialColor 

if pPocket- >Color was changed 

set pPocket- >f Paint to TRUE 
set f InternalEvent to TRUE 
if pPocket- >Color was changed from a value greater 
30 than or equal to pPocket- >CrestTide to a value 

below pPocket- >CrestTide 

set Event to EV_STEP_DOWN 

else 

set Event to EV NULL 
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break 

case ST_DWELL: 

/* set state to previous 
5 state */ 

/* in preparation for the 
next */ 

/* state transition */ 
set pPocket->State to pPocket- >PreviousState 
10 increment pPocket - >Color by pPocket - >Increment , 

but not above pPocket- >Ceiling 
if pPocket- >Color was changed 
set pPocket ->f Paint to TRUE 
set f InternalEvent to TRUE 
15 if pPocket- >Color was changed from a value below 

pPocket- >CrestTide to a value greater than or 
equal to pPocket- >CrestTide 
set Event to EV_STEP_UP 

else 

20 set Event to EV_NULL 

break 

case ST_IDLE: 

set the system cursor location to the center of 
25 the display 

set f InternalEvent to TRUE 
set Event to EV_NULL 
break 

30 case ST_EBB_TIDE : 

break 

case ST_ENTRY : 

set f InternalEvent to TRUE 
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set Event to EVNULL 
break 

case ST_DISCARD: 

/* set state to previous 
state */ 

/* for the next transition*/ 
set pPocket - >State to pPocket - >PreviousState 
break 

case ST_EXIT: 

set Event to EV_NULL 

set f InternalEvent to TRUE 

if pPocket- >f Interior equals TRUE 

change the boundaries of the Windows® region 
having the handle pPocket- >hRegi on to those 
defined by the second array of points 
associated with this state machine 

set pPocket->f Paint to TRUE 

send EV_RESET to the state machine having the 

index pPocket- >iAdj acentPocket 

break 

default : 

display error message 
break 

} 

/* end switch */ 

} 

/* end while */ 

if pPocket - >f Paint is equal to TRUE 

invalidate the entire client area 

} 
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Procedure : CreateEvent 



/***************************************************+*************/ 



/* */ 

5 /* Procedure: CreateEvent */ 

/* */ 

/+ Description: Examines movement of cursor and determines */ 

/* what events have occurred. */ 

/* */ 

10 /* Input Parameters: */ 

/* pEvent pointer to an event data structure */ 

/* for output */ 

/* Output: A completed event data structure */ 

/* indicated by pEvent including: */ 

15 /* (1) an indication of whether the cursor hotspot */ 

/* intersects a selectable region */ 

/* (2) the index of the intersected selectable */ 

/* region, if any */ 

/* (3) the event for the intersected selectable */ 

20 /* region, if any */ 

/* (4) the event for all non- intersected */ 

/* selectable regions */ 

/* */ 

/* Updated variables indicate: */ 

25 /* (1) the previous location of the cursor hotspot */ 

/* (2) the approximate time of the sampling of the */ 

/* previous location of the cursor hotspot */ 

/* */ 



/*********************************************^ 

30 CreateEvent (pEvent) 

{ 

get current cursor hotspot location 

if there has been cursor hotspot movement since the last 
cursor location 
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sampling 

set time of previous cursor movement to time latest 
received 

WM_TIMER message was generated 

if cursor hotspot has crossed out of a selectable region 
set event to EV_CROSS_OUT for the intersected 
selectable region and to EV_MOVEMENT for all other 
selectable regions 

else 

if cursor hotspot intersects a selectable region 

if intersected selectable region's Color equals 
Ceiling 

set event to EV_CEILING for the intersected 
selectable region and to EV_DECAY for all 
other selectable regions 

else 

set event to EV_DWELL for the intersected 
selectable region and to EV_DECAY for all 
other selectable regions 

else 

set event to EV_MOVEMENT for all selectable 
regions 

else 

if cursor hotspot intersects a selectable region 
if idle time exceeded 

set event to EV_IDLE_TIMEOUT for all selectable 
regions 

else 

if intersected selectable region's Color equals 
Ceiling 

set event to EV_CEILING for the intersected 
selectable region and to EV_DECAY for all 
other selectable regions 

else 
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set event to EV_DWELL for the intersected 
selectable region and to EV_DECAY for all 
other selectable regions 

else 

5 set event to EV_DECAY for all selectable regions 

set the previous cursor hotspot location to the current 
hotspot location 

} 

10 /* end CreateEvent */ 
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Claims 

I claim: 

1 . For use with a human interface system wherein a body member of an operator may indicate a 
first location on a surface, the surface including a display area, a method of selecting a 
sequence of one or more graphic symbols from a plurality of sequences of one or more 
graphic symbols, one or more sequences of the plurality of sequences including one or more 
ideographs, said method comprising the steps of: 

displaying on the display area a plurality of selectable regions, the plurality of selectable 
regions together at least partially circumscribing a region on the display area, at least two of 
the selectable regions associated respectively with one sequence of the plurality of 
sequences, each of the plurality of sequences hav ing a first common characteristic: and 

in response to a selection event wherein the first location indicated by the body member at or 
near the time the selection event occurs intersects any one of the at least two selectable 
regions, selecting the sequence associated with the intersected selectable region. 

2. The method of claim 1 wherein each of the sequences belongs to any one of the Chinese 
language, Japanese language, Korean language, Picture Communication Symbols symbol set, 
Rebus symbol set, Picsyms symbol set, Pictogram Ideogram Communication Symbols 
symbol set, Yerkish symbol set. Blissymbolics symbol set. and a set of depictions of the 
signs of a manual sign language. 

3. The method of claim 1 further comprising confining the first location indicated by the body 
member of the operator to the display area. 

4. The method of claim 1 wherein the operator keeps at least five of his fingers, not including 
either thumb, on the home row of a keyboard while moving the body member to indicate the 
first location on the surface. 

5. The method of claim 1 wherein the first common characteristic includes any one of: 
(a) a phonetic unit, including either one of a Chinese pronunciation and a Japanese 

pronunciation; 
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(b) 


an intonation; 


(c) 


a stroke used to draw an ideograph; 


(d) 


a number of horizontal strokes; 


(e) 


a number of vertical strokes; 


(0 


a number of total strokes; 


(g) 


a stroke order; 


(h) 


a radical; 


(i) 


a part of speech; 


Ci) 


an ideograph; 


(k) 


a kana; 


(1) 


a diacritic; 


(m) 


a classification of a part of an ideograph; 


(n) 


a meaning; and 


(o) 


a meaning class. 



15 

6. The method of claim 1 wherein the body member of the operator is a body member other 
than the shoulder, arm, elbow, wrist, hand, finger, or thumb. 



7. The method of claim 6 wherein the body member of the operator is the head of the operator. 

20 

8. The method of claim 1 wherein the displaying step further comprises displaying a first cursor 
on the display area at the first location indicated by the body member; and wherein the 
selecting step further comprises removing the first cursor from the display area. 



25 9. The method of claim 8 wherein the selecting step further comprises displaying a second 
cursor at a predetermined location on the display area. 

10 The method of claim 1 further comprising displaying on the display area one sequence of the 
plurality of sequences associated with one of the at least two selectable regions. 



30 



1 1 . The method of claim 10 wherein the selecting step further comprises displaying a first cursor 
on the display area at a location on or near the displayed sequence. 

12. The method of claim 10 wherein the step of displaying one sequence of the plurality of 
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sequences further comprises indicating the association of the displayed sequence with its 
associated selectable region. 

13. The method of claim 1 2 wherein the indicating step further comprises displaying any part of 
either of: 

(a) the foreground of the displayed sequence; and 

(b) the background of the displayed sequence 



in any one 


of: 


(1) 


a location: 


(2) 


a size; 


(3) 


a shape; 


(4) 


a hue; 


(5) 


a brightness; 


(6) 


a contrast; 


(7) 


a tone; 


(8) 


a dithering; 


(9) 


a fill; and 


(10) 


a font 



associated with the associated selectable region. 

14. The method of claim 1 further comprising selecting the first common characteristic from a 
plurality of common characteristics of ideographs. 

15 The method of claim 1 wherein the selection event includes a switch operation. 

16. The method of claim 1 wherein at least one sequence of the plurality of sequences includes 
one or more graphic symbols in addition to the graphic symbols of the at least one sequence 
representing the first common characteristic. 

1 7. The method of claim 1 6 wherein the first common characteristic includes an ideograph and 
the additional graphic symbols of the at least one sequence include any one of: 

(a) a prefix; 
(b an infix: and 
(c) a suffix; 
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which may be added to the ideograph. 

1 8. The method of claim 1 wherein the at least two of the selectable regions are ordered 
responsive to any one of: 

(a) the frequency of use of the associated sequence in an ideographic language; 

(b) the frequency of use of the associated sequence in a predetermined operator's usage; 

(c) the recency of use of the associated sequence in a predetermined operator's usage; 

(d) the length of associated sequence; 

(e) the alphabetic order of the associated sequence; and 

(f) a second common characteristic. 

1 9. For use with a human interface system including a keyboard wherein a body member of an 
operator may indicate a first location on a surface, the surface including a display area, a 
method of selecting a sequence of one or more graphic symbols from a plurality of sequences 
of one or more graphic symbols, one or more sequences of the plurality of sequences 
including one or more ideographs, said method comprising the steps of: 

displaying on the display area a plurality of selectable regions, at least two of the selectable 
regions associated respectively with one sequence of the plurality of sequences, each of the 
plurality of sequences having a common characteristic; 

moving the body member to indicate the first location on the surface while at least five of the 
operator's fingers, not including either thumb, remain on the home row of the keyboard; and 

in response to a selection event wherein the first location indicated by the body member at or 
near the time the selection event occurs intersects any one of the at least two selectable 
regions, selecting the sequence associated with the intersected selectable region. 

20. The method of claim 1 or 19 wherein the body member may indicate a second location on the 
surface and wherein the selection event includes the duration of a period of intersection 
equalling or exceeding a selection threshold period, the period of intersection starting in 
response to the first location intersecting any one of the selectable regions and ending in 
response to the second location intersecting the selectable region intersected by the first 
location. 
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2 1 . For use with a human interface system wherein a body member of an operator may indicate a 
location outside a display area, a method of selecting a sequence of one or more graphic 
symbols from a plurality of sequences of one or more graphic symbols, one or more 
5 sequences of the plurality of sequences including one or more ideographs, said method 

comprising the steps of: 



at least partially delimiting a plurality of selectable regions outside the display area, at least 
two of the selectable regions associated respectively with one sequence of the plurality of 
10 sequences, each of the plurality of sequences having a common characteristic: and 

in response to a selection event wherein the location indicated by the body member at or near 
the time the selection event occurs intersects any one of the at least two selectable regions, 
selecting the sequence associated with the intersected selectable region. 

15 

22. The method of claim 21 wherein the body member of the operator is a body member other 
than either of the operator's eyes. 



23. For use with a human interface system wherein a body member of an operator may indicate 
20 successive locations on a surface, the surface including a display area, a method of selecting 

a sequence of one or more graphic symbols from a plurality of sequences of one or more 
graphic symbols, one or more sequences of the plurality of sequences including one or more 
ideographs, said method comprising the steps of: 

25 at least partially delimiting a plurality of selectable regions, each of the selectable regions 

including an invisible subregion adjacent the display area and a visible subregion on the 
display area, the plurality of visible subregions together at least partially circumscribing a 
region on the display area, at least two of the selectable regions associated respectively with 
one sequence of the plurality of sequences, each of the plurality of sequences having a 

30 common characteristic; and 



responsive to the duration of a period of intersection equalling or exceeding a selection 
threshold period, the period of intersection starting in response to a starting location 
intersecting any one of the invisible subregions included in one of the at least two selectable 
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regions and ending in response to an ending location intersecting the invisible subregion 
intersected by the starting location, the starting and the ending locations each being one of 
the successive locations, selecting the sequence associated with the selectable region 
including the invisible subregion intersected by the starting location. 

5 

24. The method of claim 23 further comprising indicating the duration of a period of 
intersection, said period starting in response to the starting location and ending in response to 
an intermediate location intersecting the invisible subregion intersected by the starting 
location, the intermediate location being one of the successive locations occurring after the 

10 starting location. 

25. For use with a human interface system wherein a body member of an operator may indicate 
successive locations on a surface, the surface including a display area, a method of selecting 
a sequence of one or more graphic symbols from a plurality of sequences of one or more 

15 graphic symbols, one or more sequences of the plurality of sequences including one or more 

ideographs, said method comprising the steps of: 

displaying on the display area a plurality of selectable regions, the plurality of selectable 
regions together at least partially circumscribing a region on the display area, at least two of 
20 the selectable regions associated respectively with one sequence of the plurality of 

sequences, each of the plurality of sequences having a common characteristic; and 

responsive to a quantity equalling or exceeding a selection threshold period, said quantity 
being a function of the durations of a first and a second period of intersection 

25 

said First period of intersection starting in response to a first starting location 
intersecting any one of the at least two selectable regions and ending in response to a 
first ending location intersecting the selectable region intersected by the first starting 
location, 

30 

said second period of intersection starting in response to a second starting location 
intersecting the selectable region intersected by the first starting location and ending in 
response to a second ending location intersecting the selectable region intersected by 
the first starting location. 
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the first starting location, the second starting location, the first ending location and the 
second ending location each comprising one of the successive locations. 



selecting the sequence associated with the selectable region intersected by the first starting 
location. 

26. The method of claim 25 further comprising increasing the selection threshold period 
responsive to the duration of an intermediate period of non-intersection of the selectable 
region intersected by the first starting location, said intermediate period of intersection 
starting in response to an intermediate starting location not intersecting the selectable region 
intersected by the first starting location and ending in response to an intermediate ending 
location not intersecting the selectable region intersected by the first starting location, the 
intermediate starting and the intermediate ending locations each comprising one of the 
successive locations and each occurring at the time or after the first ending location occurs 
and before or at the time the second starting location occurs. 

27. The method of claim 26 further comprising indicating the duration of the intermediate period 
of non-intersection. 



28. A method of selecting a sequence of one or more graphic symbols from a plurality of 
sequences of one or more graphic symbols, one or more sequences of the plurality of 
sequences including one or more ideographs, said method comprising the steps of: 

displaying on a display a plurality of selectable regions, each of the selectable regions 
associated respectively with one sequence of the plurality of sequences, each of the plurality 
of sequences having a common characteristic; 

receiving a movement related signal and moving a cursor on the display responsive thereto; 
and 

selecting the sequence associated with the particular one of the plurality of selectable regions 
most nearly along a path indicated by a cursor location and the movement related signal in 
advance of an intersection of the cursor and said particular selectable region. 
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29. A method of selecting a sequence of one or more graphic symbols from a plurality of 
sequences of one or more graphic symbols, one or more sequences of the plurality of 
sequences including one or more ideographs, said method comprising the steps of: 

5 

displaying on a display a plurality of selectable regions, each of the selectable regions 
associated respectively with one sequence of the plurality of sequences, each of the plurality 
of sequences having a common characteristic; 

10 receiving a movement related signal and moving a cursor on the display responsive thereto; 

and 

responsive to: 

(a) an intersection of the cursor and a selectable region: and 
15 (b) a cursor path toward the intersected selectable region, 

selecting the sequence associated with the intersected selectable region. 

30. The method of claim 29 wherein the selecting step is further responsive to the duration of the 
20 period of the intersection equalling or exceeding a selection threshold period. 

3 1 . For use with a human interface system wherein a body member of an operator may indicate 
successive locations on a surface, the surface including a display area, a method of selecting 
an undo function, said method comprising the steps of: 

25 

first displaying on the display area a plurality of selectable regions, the plurality of selectable 
regions together at least partially circumscribing a region on the display area, at least two of 
the selectable regions associated respectively with one sequence of a plurality of sequences 
of graphic symbols, one or more sequences of the plurality of sequences including one or 
30 more ideographs and each having a common characteristic; 

in response to a first selection event wherein a first one of the successive locations indicated 
by the body member at or near the time the first selection event occurs intersects any one of 
the at least two selectable regions, selecting the sequence associated with the intersected 
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selectable region; 

performing a function with the selected sequence; 

5 second displaying on the display area a selectable region associated with the undo function; 

and 

in response to a second selection event wherein a second one of the successive locations 
indicated by the body member at or near the time the second selection event occurs intersects 
10 the selectable region associated with the undo function, undoing the function performed with 

the selected sequence. 

32. A method of data entry in an ideographic language, said method comprising the steps of: 

15 displaying on a display a plurality of selectable regions, the plurality of selectable regions 

together at least partially circumscribing a region on the display, at least two of the selectable 
regions associated respectively with one sequence of a plurality of sequences of graphic 
symbols, one or more sequences of the plurality of sequences including one or more 
ideographs and each having a common characteristic; 

20 

receiving a movement related signal and moving a cursor on the display responsive thereto: 

responsive to a cursor path toward any one of the at least two selectable regions, changing a 
selection threshold period associated or to be associated with said selectable region: and 

25 

responsive to the duration of a period of intersection of the cursor and said selectable region 
equalling or exceeding the selection threshold period, entering into the at least partially 
circumscribed region the sequence associated with said selectable region. 

30 33 . For use with a speech synthesis system including a speech synthesizer and a display, a 
method of speaking comprising the steps of: 

displaying on the display a plurality of selectable regions, the plurality of selectable regions 
together at least partially circumscribing a region on the display, at least two of the selectable 
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regions associated respectively with one sequence of a plurality of sequences of graphic 
symbols, one or more sequences of the plurality of sequences including one or more 
ideographs and each having a common characteristic; 

5 moving a cursor on the display responsive to the movement of a body member of an 

operator; and 

in response to a selection event wherein the cursor at or near the time the selection event 
occurs intersects any one of the at least two selectable regions, speaking, by means of the 
10 speech synthesizer, a word associated with the sequence associated with the intersected 

selectable region. 

34. An apparatus for selecting a menu option from a plurality of menu options, said apparatus 
comprising: 

15 (a) display means for displaying on a display the plurality of menu options, each of the 

plurality of menu options associated respectively with a sound; 

(b) sound receiving means for receiving a sound signal; 

(c) match means for matching the received sound signal to any one of the plurality of 
sounds; and 

20 (d) selection means for selecting the menu option associated with the matched sound. 

35. The apparatus of claim 34 wherein one or more of the menu options represents a sequence of 
one or more letters and the sound associated w ith each of the one or more menu options is 
not a phonetic representation of the menu option. 

25 

36. The apparatus of claim 34 wherein each of the plurality of sounds is associated respectively 
with a sound indicator other than the displayed menu option. 

37. The apparatus of claim 36 further comprising indication means for . indicating the association 
30 of each sound indicator to the menu option associated with the sound associated with the 

sound indicator. 

38. The apparatus of claim 37 wherein each of the sound indicators is outside the display. 
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39. The apparatus of claim 36 wherein one or more of the sound indicators includes a non- 
alphanumeric indicator. 

40. The apparatus of claim 39 wherein the non-alphanumeric indicator includes any one of a hue 
5 indicator, a pitch indicator, a volume indicator, a sound duration indicator, a change in pitch 

indicator, a change in volume indicator, a shape indicator, and an object indicator. 

41 . For use with a general purpose computer system including a display, an apparatus for 
selecting a menu option from a plurality of menu options, said apparatus comprising: 

10 (a) display means for displaying on the display the plurality of menu options, each of the 

plurality of menu options associated respectively with a hue displayed intersecting or 
near the displayed menu option; 

(b) sound receiving means for receiving a signal representing a sound: 

(c) match means for matching the received sound to a name of any one of the plurality of 
15 hues; and 

(d) selection means for selecting the menu option associated with the hue corresponding to 
the matched name. 

42. The apparatus of claim 34 or 41 wherein the plurality of displayed menu options together at 
20 least partially circumscribe a region on the display. 

43. An apparatus for selecting a menu option from a plurality of menu options, said apparatus 
comprising: 

(a) a surface on which may be displayed the plurality of menu options, each of the 

25 plurality of menu options associated respectively with a sound indicator other than the 

displayed menu option; 

(b) a sound matcher for matching a sound to any one of the plurality of sound indicators; 
and 

(c) a selector for selecting the menu option associated with the matched sound indicator. 



30 



44. For use with a computer capable of executing an application program, a computer access 
device, said device comprising: 

(a) a menu hierarchy including a menu including a plurality of menu options, one or more 
of the menu options associated respectively with a sound which is not a phonetic 
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representation of the menu option and with a submenu including a plurality of 
submenu options, each of the submenu options associated respectively with a sound 
which is not a phonetic representation of the submenu option and with a sequence of 
one or more characters; 

(b) sound receiving means for receiving a first and a second sound signal; 

(c) match means for matching the first sound signal to any one of the sounds associated 
with the one or more menu options and for matching the second sound signal to any 
one of the sounds associated with the submenu options; and 

(d) input means, responsive to the match means matching the first and the second sound 
signals, for inputting to the application program the sequence of one or more 
characters associated with the submenu option associated with the sound matched by 
the second sound signal. 

45. The computer access device of claim 44 further comprising: 

(e) a display: and 

(f) means for displaying the one or more menu options on the display, and, responsive to 
the match means matching the first sound signal, displaying the submenu options on 
the display. 

46. For use with a general purpose computer system including a display on which may be 
displayed a plurality of menu options; and including sound receiving means for receiving a 
sound signal, an apparatus for selecting a menu option from the plurality of menu options, 
said apparatus comprising: 

(a) a medium readable by the general purpose computer system; and 

(b) a program, stored on the medium and executable by the general purpose computer 

system, for: 

(1 ) displaying on the display the plurality of menu options, each of the plurality of 
menu options associated respectively with a sound which is not a phonetic 
representation of the menu option; 

(2) matching the received sound signal to any one of the plurality of sounds; and 

(3) selecting the menu option associated with the matched sound. 

47. The apparatus of claim 46 wherein the medium includes a random access memory . 
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48. The apparatus of claim 46 wherein the medium is coupled to the general purpose computer 
system. 



49. The apparatus of claim 46 wherein the medium may be coupled to and uncoupled from the 
general purpose computer system. 

50. The apparatus of claim 46 wherein the medium includes either one of a magnetic store and 
an optical store. 

51. A speech synthesis system comprising: 

(a) a display for displaying a plurality of menu options, each of the menu options 
associated respectively with a sound and with a sequence of one or more characters; 

(b) sound receiving means for receiving successive sound signals; 

(c) a speech synthesizer: and 

(d) control means for 

(1) matching one of the successive sound signals to any one of the plurality of 
sounds; 

(2) repetitively 

(I) selecting the sequence of one or more characters associated with the menu 

option associated with the matched sound; and 
(ii) appending the selected sequence to at least one previously selected 

sequence; and 

(3) speaking, by means of the speech synthesizer, the word spelled by the appended 
sequences. 

52. A method of selecting a sequence of one or more graphic symbols from a plurality of 
sequences of one or more graphic symbols, one or more sequences of the plurality of 
sequences including one or more ideographs, said method comprising the steps of: 

displaying on a display the plurality of sequences, each of the plurality of sequences having a 
common characteristic and each associated respectively with a sound indicator; 

matching a sound to any one of the plurality of sound indicators: and 
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selecting the sequence of the plurality of sequences associated with the matched sound 
indicator. 



53. The method of claim 52 wherein the sound is not a phonetic representation of the sequence 
associated with the matched sound indicator. 

54. The method of claim 52 wherein the sound indicator is non-alphanumeric. 

55. The method of claim 54 wherein the sound indicator is a hue. 

56. The method of claim 52 further comprising the step of indicating the association of each of 
the plurality of sequences with the associated sound indicator. 

57. The method of claim 52 wherein each of the sequences belongs to any one of the Chinese 
language, Japanese language, Korean language, Picture Communication Symbols symbol set. 
Rebus symbol set, Picsyms symbol set, Pictogram Ideogram Communication Symbols 
symbol set, Yerkish symbol set, Blissymbolics symbol set. and a set of depictions of the 
signs of a manual sign language. 

58. An apparatus for selecting a menu option from a plurality of menu options, said apparatus 
comprising: 

(a) a display area; 

(b) means for at least partially delimiting a plurality of selectable regions, each of the 
selectable regions outside the display area and each associated respectively with a 
menu option; 

(c) movement related signal receiving means for receiving a movement related signal 
indicating a location with respect to the display area; and 

(d) selection means for selecting, in response, to a selection event associated with the 
selectable region intersected by the location, the menu option associated with the 
intersected selectable region. 

59. The apparatus of claim 58 wherein the selection event includes a switch operation at or near 
the time the intersection occurs. 
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60. The apparats of claim 58 further comprising means for indicating the menu option associated 
with each selectable region. 

61 . The apparatus of claim 58 further comprising means for indicating which one of the 
5 selectable regions is intersected by the location. 

62. The apparatus of claim 58 further comprising location indication means for indicating the 
location of each selectable region. 

10 63. The apparatus of claim 62 wherein the location indication means further comprises means for 
displaying each menu option on the display area, wherein the location of each displayed 
menu option indicates the location of the associated selectable region. 

64 The apparatus of claim 62 where the location indication means includes means for displaying 
15 at least part of each selectable region on the display area. 

65. The apparatus of claim 58 wherein the selection event includes a switch operation; and 
wherein the selection means further comprises switch operation receiving means for 
receiving a signal indicating the switch operation. 

20 

66. The apparatus of claim 58 wherein the selection means is at least partially disabled in 
response to a first selection event. 

67. The apparatus of claim 66 wherein the selection means, in response to a second selection 
25 event, is restored to the functionality it had prior to the first selection event. 

68. The apparatus of claim 58 wherein all or all but one of the selectable regions are partially 
delimited. 

30 69. The apparatus of claim 68 wherein one of the selectable regions is completely delimited. 

70. The apparatus of claim 68 further comprising a computer system including display means for 
displaying at least part of the output of an application program executable on the computer 
system in the region on the display area and wherein at least one of the menu options 
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represents an input to the application program. 



71 . An apparatus for selecting a menu option from a plurality of menu options, said apparatus 
comprising: 

(a) a display area; 

(b) means for at least partially delimiting a plurality of selectable regions, each of the 
selectable regions outside the display area and each associated respectively with a 
menu option; 

(c) movement related signal receiving means for receiving a movement related signal 
indicating successive locations with respect to the display area; and 

(d) selection means, responsive to a first dwell event associated with any one of the 
selectable regions outside the display area, for selecting the menu option associated 
with the selectable region outside the display area associated with the first dwell event. 

72. The apparatus of claim 71 further comprising a pointer, responsive to the movement of a 
body member of an operator other than either of the operator's eyes, for generating the 
movement related signal. 

73. The apparatus of claim 71 wherein the selection means is responsive to a plurality of periods 
of intersection, each of two or more of the successive locations and the intersected selectable 
regions. 

74. The apparatus of claim 71 wherein some of the selectable regions are not completely visible. 

75. The apparatus of claim 71 wherein at most one of the selectable regions is adjacent the 
display area. 

76. The apparatus of claim 71 wherein each of the successive locations is relative to a 
predetermined location on the display area or to a previous location of the successive 
locations. 



77. The apparatus of claim 71 wherein the first dwell event includes a first quantity equalling or 
exceeding a predetermined quantity, the first quantity being a function of the durations of 
one or more successive periods of intersection of two or more of the successive locations and 
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one of the selectable regions; and wherein the selectable region associated with the first 
dwell event is the intersected selectable region. 

78. The apparatus of claim 71 wherein the first dwell event includes a first quantity equalling or 
5 exceeding a predetermined quantity, the first quantity being a function of: 

( 1 ) the durations of one or more successive periods of intersection of two or more of the 
successive locations and a first one of the selectable regions: and 

(2) the durations of one or more successive periods of intersection of two or more of the 
successive locations and a second one of the selectable regions: and 

10 wherein the selectable region associated with the first dwell event is the selectable region 

intersected by one of the successive locations when the first dwell event occurs. 

79. The apparatus of claim 71 further comprising a plurality of selectable regions on the display 
area . each associated respectively with one of the selectable regions outside the display area: 

15 and wherein the selection means is further operative, responsive to a second dwell event 

associated with any one of the selectable regions on the display area, to select the menu 
option associated with the selectable region outside the display area associated with the 
selectable region on the display area associated with the second dwell event. 

20 80. The apparatus of claim 79 wherein the selection means is further operative, responsive to a 
first quantity equalling or exceeding a predetermined quantity, the first quantity being a 
function of: 

(a) the durations of one or more successive periods of intersection of two or more of the 
successive locations and one of the selectable regions on the display area: and 
25 (b) the durations of one or more successive periods of intersection of two or more of the 

successive locations and the selectable region outside the display area associated with 
the intersected selectable region on the display area; 

to select the menu option associated with the selectable region outside the display area. 

30 81. The apparatus of claim 79 wherein each of one or more of the selectable regions on the 
display area is adjacent the associated selectable region outside the display area. 

82. The apparatus of claim 79 wherein each of one or more of the selectable regions on the 

display area indicates the location of the associated selectable region outside the display area. 
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83. The apparatus of claim 79 wherein the plurality of selectable regions on the display area 
together at least partially circumscribe a region on the display area. 

5 84. The apparatus of claim 83 wherein the selection means further includes means for indicating 
the remaining dwell time required to select the intersected selectable region. 

85. The apparatus of claim 83 w herein the movement related signal is responsive to the 
movement of a body member of an operator having impaired ability to sense the position of 

10 the body member and the apparatus further comprises tactile indication means for indicating 

tactilely to the operator the position of the body member. 

86. The apparatus of claim 83 wherein the selection means further includes means for indicating 
on the display area the location of one of the successive locations located outside the display 

15 area. 

87. The apparatus of claim 83 wherein the selection means further includes means for indicating 
on the display area the distance between one of the successive locations located outside the 
display area and the point on the display closest thereto. 

20 

88. In a human interface system including a display whereon a first cursor may be displayed and 
moved responsive to successive locations indicated by a movement related signal, an 
apparatus for selecting a menu option associated with an overshot selectable region on the 
display, said apparatus comprising: 

25 (a) means for displaying a plurality of selectable regions within a first polygon 

intersecting the display, each selectable region associated respectively with a menu 
option, each selectable region adjacent a side of the first polygon and the plurality of 
selectable regions together at least partially circumscribing a region on the display; 

(b) movement related signal receiving means for receiving the movement related signal 
30 indicating the successive locations; and 

(c) control means for 

( 1 ) moving the first cursor within the first polygon responsive to the successive 
locations indicated by the movement related signal; 

(2) confining at least part of the first cursor to the first polygon: and 
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(3) in response to a first quantity equalling or exceeding a predetermined quantity, 
the first quantity being a function of the durations of one or more successive 
periods of intersection of the first cursor and one of the selectable regions, 
selecting the menu option associated with the intersected selectable region. 

89. The apparatus of claim 88 wherein the first polygon is located on the display. 

90. The apparatus of claim 89 wherein at least one of the selectable regions intersects the at least 
partially circumscribed region. 

9 1 . The apparatus of claim 89 wherein the control means is further operative to confine at least 
part of the first cursor to a second polygon on the display. 

92. The apparatus of claim 91 wherein the control means is further operative to switch, 
responsive to an intersection of the first cursor and one of the selectable regions, from 
confining at least part of the first cursor to the first polygon to confining at least part of the 
first cursor to the second polygon . 

93. The apparatus of claim 91 wherein the control means is further operative to switch, 
responsive to a distance between two of the successive locations, from confining at least part 
of the first cursor to the first polygon to confining at least part of the first cursor to the 
second polygon 

94. The apparatus of claim 91 wherein the control means is further operative to switch, 
responsive to an angle indicated by three of the successive locations, from confining at least 
part of the first cursor to the first polygon to confining at least part of the first cursor to the 
second polygon 



95. The apparatus of claim 91 wherein the first polygon intersects the second polygon. 

96. The apparatus of claim 95 wherein the first polygon includes all of the area of the second 
polygon. 
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97. The apparatus of claim 89 wherein the selection of the control means is further responsive to 
the proximity of one of the successive locations indicated by the movement related signal to 
the location of the first cursor. 

98. The apparatus of claim 89 wherein the selection of the control means is further responsive to 
the proximity of one of the successive locations indicated by the movement related signal to 
the intersected selectable region. 

99. The apparatus of claim 89 wherein the first polygon has at least five sides. 

100. The apparatus of claim 89 wherein at least one of the plurality of selectable regions is 
associated with an icon on the display. 

101. The apparatus of claim 100 wherein the icon represents one of a sign of a manual sign 
language, a location relative to a human body, a movement of a manual sign language a topic 
of conversation, a sentence, a desired direction of movement of a second cursor on the 
display, a sequence of one or more graphics including an ideograph, and a symbol of a 
symbol set. 

102. An apparatus for selecting a submenu option from a menu hierarchy, said apparatus 
comprising: 

(a) a display area; 

(b) a menu comprising a plurality of menu options, at least one of the menu options 
associated with a submenu comprising a plurality of submenu options; 

(c) means for at least partially delimiting: 

(1) a plurality of first selectable regions, each of the first selectable regions 
associated respectively with one of the menu options and each of the first 
selectable regions including a first subregion adjacent the display area and a first 
subregion on the display area, the plurality of the first subregions on the display 
area together at least partially circumscribing a first region on the display area; 
and 

(2) a plurality of second selectable regions, each of the second selectable regions 
associated respectively with one of the submenu options and each of the second 
selectable regions including a second subregion adjacent the display area and a 
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second subregion on the display area, the plurality of the second subregions on 
the display area together at least partially circumscribing a second region on the 
display area; 

(d) movement related signal receiving means for receiving a movement related signal 
5 indicating successive locations with respect to the display area; and 

(e) selection means for selecting, in response to a first dwell event, the menu option 
associated with the first selectable region intersected by one of the successive locations 
indicated by the movement related signal, the menu option being one of the menu 
options associated with a submenu, and for selecting, in response to a second dwell 

10 event, the submenu option associated with the second selectable region intersected by 

one of the successive locations indicated by the movement related signal. 

103. The apparatus of claim 102 wherein one of the menu options represents a group of characters 
and wherein a first one of the submenu options represents a first one character of the group 

15 of characters. 

104. The apparatus of claim 103 wherein each character of the group of characters has one of: 
(a) an extension at least a predetermined distance above the baseline of the group of 

characters; 

20 (b) an extension below the baseline of the group of characters; 

(c) lack of the characteristic described in (a); and 

(d) lack of the characteristic described in (b). 



25 105. The apparatus of claim 1 03 wherein the distance on the display area between the first 

subregion on the display area associated with the menu option representing the group of 
characters and the second subregion on the display area associated with the submenu option 
representing the first one character of the group of characters is responsive to the frequency 
of use of the first one character. 

30 

106. The apparatus of claim 103 wherein: 

(a) a second one of the submenu options represents a second one character of the group of 
characters; 

(b) the first one character is more frequently used than the second one character; and 
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(c) the distance on the display area between the first subregion on the display area 

associated with the menu option representing the group of characters and the second 
subregion associated with the submenu option representing the first one character of 
the group of characters is less than the distance on the display area between the first 
subregion on the display area associated with the menu option representing the group 
of characters and the second subregion on the display area associated with the 
submenu option representing the second one character of the group of characters. 



107. The apparatus of claim 1 03 wherein the position of a character of the group of characters 
indicates the position of the second subregion on the display area associated with the 
submenu option representing the first one character of the group of characters. 

108. An apparatus for selecting a menu option from a plurality of menu options, said apparatus 
comprising: 

(a) a surface; 

(b) means for delimiting a plurality of selectable regions on the surface, each of the 
selectable regions associated respectively with a menu option, the plurality of 
selectable regions together at least partially circumscribing a region on the surface; 

(c) a pointer, responsive to the movement of a one of an operator's limbs, digits and head, 
for indicating successive locations on the surface; and 

(d) selection means for selecting, in response to a dwell event, the menu option associated 
with the selectable region intersected by one of the successive locations indicated by 
the pointer. 

109. An apparatus for selecting a menu option from a plurality of menu options, said apparatus 
comprising: 

(a) means for displaying a plurality of selectable regions on a display area, each of the 
selectable regions associated respectively with a menu option, the plurality of 
selectable regions together at least partially circumscribing a region on the display 
area; 

(b) movement related signal receiving means for receiving a movement related signal 
indicating successive locations on the surface; and 

(c) in response a quantity equalling or exceeding a predetermined quantity, the quantity 
being a function of the durations of a plurality of successive periods of intersection of 
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two or more of the successive locations and one of the selectable regions, selection 
means for selecting the menu option associated with the intersected selectable region. 

apparatus for selecting an option from a menu, said apparatus comprising: 

cursor movement means for receiving a movement related signal and for moving a 
cursor on a display responsive to the received movement signal; 
delimit means for delimiting on the display a first plurality of regions and a second 
plurality of selectable regions, each of the second plurality of selectable regions 
associated respectively with a menu option: the first plurality of regions together at 
least partially circumscribing a first region on the display; and 
selection means, responsive only to an intersection of the cursor and a first one of the 
first plurality of regions and thereafter to a first selection event associated with one of 
the second plurality of selectable regions, for selecting the menu option associated with 
the selectable region associated with the first selection event. 

111. The apparatus of claim 1 1 0 further comprising means for receiving a switch operation signal; 
and wherein the delimit means includes means for displaying the first plurality of regions 
responsive to the received switch operation signal. 

1 1 2. The apparatus of claim 1 1 0 wherein the second plurality of selectable regions together at 
least partially circumscribing the first region on the display. 

1 1 3 The apparatus of claim 1 1 0 further comprising a third plurality of selectable regions, each of 
the third plurality of selectable regions associated respectively with a menu option; and 
wherein the selection means is further responsive to an intersection of the cursor and a 
second one of the first plurality of regions and thereafter to a second selection event 
associated with one of the third plurality of selectable regions, for selecting the menu option 
associated with the selectable region associated with the second selection event. 

1 14. The apparatus of claim 1 1 0 wherein the selection means includes means for receiving a 
switch operation signal; and wherein the first selection event includes: 

(1 ) an intersection of the cursor and the selectable region associated with the second 
selection event: and 

(2) at or near the time the intersection occurs, receipt of the switch operation signal. 
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115. In a human interface system wherein a body member of an operator may indicate successive 
locations with respect to a display, a menu option selector for selecting a menu option from a 
plurality of menu options, said menu option selector comprising: 

(a) the display having thereon a first plurality of selectable regions, each of the first 
plurality of selectable regions associated respectively with one of the menu options; 

(b) means for at least partially delimiting a second plurality of selectable regions, each of 
the second plurality of selectable regions located outside the display and each 
associated respectively with one of the first plurality of selectable region: 

(c) in response to a first quantity equalling or exceeding a predetermined quantity, the first 
quantity being a function of: 

( 1 ) the durations of one or more successive periods of intersection of two or more of 
the successive locations and one of the selectable regions on the display area; 
and 

(2) the durations of one or more successive periods of intersection of two or more of 
the successive locations and the selectable region outside the display area 
associated with the one of the selectable regions on the display area; 

a selector for selecting the menu option associated with the one of the selectable 
regions on the display area. 

1 1 6. The apparatus of claim 1 1 5 wherein the plurality of the subregions on the display together at 
least partially circumscribing a region on the display. 

117. In a human interface system wherein a body member of an operator may indicate a location 
on a surface, a menu option selector comprising: 

(a) the surface including a display area, the display area having thereon a plurality of 
selectable regions, each of the selectable regions associated respectively with a menu 
option, the plurality of selectable regions together at least partially circumscribing a 
region on the display area; 

(b) a clipper for generating, in response to the location indicated by the body member of 
the operator indicating a location outside the display area, a clipped location indicative 
of a location on the display area: and 

(c) a selector for selecting, in response to a selection event, the menu option associated 
with the selectable region intersected by the clipped location. 
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118. In a human interface system wherein a body member of an operator may indicate a location 
on a surface, a menu option selector comprising: 

(a) the surface including a display area, the display area having thereon a plurality of 
selectable regions, each of the selectable regions associated respectively with a menu 
option, the plurality of selectable regions together at least partially circumscribing a 
region on the display area: 

(b) a confiner for confining the location indicated by the body member of the operator to 
the display area: and 

(c) a selector for selecting, in response to a selection event, the menu option associated 
with the selectable region intersected by the location indicated by the body member of 
the operator. 

119. The menu option selector of claim 1 1 7 or 1 1 8 wherein each of the plurality of selectable 
regions is adjacent an edge of the display area. 

120. In a human interface system wherein a body member of an operator may indicate successive 
locations on a surface, a menu option selector comprising: 

(a) a detector area on the surface and including a plurality of selectable regions, each of 
the selectable regions associated respectively with a menu option; 

(b) a confiner for confining the location indicated by the body member of the operator to 
the detector area; and 

(c) a selector for selecting, in response to a dwell event associated with any one of the 
selectable regions, the menu option associated with the selectable region associated 
with the dwell event. 

121. The menu option selector of claim 1 20 wherein each of the plurality of selectable regions is 
adjacent an edge of the detector area. 

122. For use with a general purpose computer system including a display on which a cursor may 
be displayed, the general purpose computer system being capable of executing an application 
program, an apparatus comprising: 

(a) a medium readable by the general purpose computer system: and 
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(b) a program, stored on the medium and executable by the general purpose computer 
system, for: 

( 1 ) displaying a plurality of selectable regions within a polygon on the display, each 
selectable region adjacent a side of the polygon, one or more of the selectable 
regions each associated respectively with a sequence of one or more characters, 
the plurality of selectable regions together at least partially circumscribing a 
region on the display: 

(2) receiving a movement related signal and moving at least part of the cursor only 
within the polygon responsive to the movement related signal; and 

(3) in response to a first quantity equalling or exceeding a predetermined quantity, 
the first quantity being a function of the durations of one or more successive 
periods of intersection of the cursor and one of the one or more selectable 
regions, inputting the sequence of one or more characters associated with the 
intersected selectable region to the application program. 



123. A data entry system including a computer system on which may be executed an application 
program, said data entry comprising: 

(a) the computer system including a display; 

(b) a pointer selected from the group consisting of a (1) mouse; (2) trackball; (3) joystick; 
(4) stylus and graphics tablet; (5) lightpen; (6) thumb wheel; (7) touch screen; (8) 

head pointer; and (9) intraoral pointer, the pointer coupled to the computer 
system: and 

(c) program means executable on the computer system for: 

(1) displaying a plurality of selectable regions within a polygon on the display, each 
selectable region adjacent a side of the polygon, the plurality of selectable 
regions together at least partially circumscribing a region on the display; 

(2) moving a cursor within the polygon responsive to movement of the pointer; and 

(3) in response to a selection event and an intersection of the cursor and a selectable 
region associated with an input for the application program, inputting the input 
to the application program. 

1 24. A computer access system for an operator having impaired motor capability, said computer 
access system including a computer system on which may be executed a computer program, 
said computer access system comprising: 
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(a) the computer system including a display: 

(b) program means executable on the computer system for: 

( 1 ) displaying a plurality of selectable regions within a polygon on the display, each 
selectable region adjacent a side of the polygon, the plurality of selectable 
regions together at least partially circumscribing a region on the display; 

(2) receiving a movement related signal and moving at least part of a cursor only 
within the polygon responsive to the movement related signal: and 

(3) in response to a selection event and an intersection of the cursor and a selectable 
region associated with an input for the computer program, inputting the input to 
the computer program. 

125. A speech synthesis system comprising: 

(a) a display on which may be displayed a plurality of selectable regions within a polygon 
on the display, each selectable region adjacent a side of the polygon and one or more 
of the selectable regions associated respectively with a sequence of one or more 
characters, the plurality of selectable regions together at least partially circumscribing 
a region on the display; 

(b) a speech synthesizer; and 

(c) control means for: 

( 1 ) receiving a movement related signal and moving a cursor within the polygon 
responsive to the movement related signal; 

(2) repetitively, in response to a selection event and an intersection of the cursor and 
one of the selectable regions associated with one of the one or more sequences 
of one or more letters, appending the sequence associated with the intersected 
selectable region to at least one previously selected sequence: and 

(3) speaking, by means of the speech synthesizer, the word spelled by the appended 
sequences. 

126. A device controller comprising: 

(a) means for displaying a plurality of selectable regions within a polygon on a surface, 
each selectable region adjacent a side of the polygon and each selectable region 
associated respectively with a device control function, the plurality of selectable 
regions together at least partially circumscribing a region of the polygon; 
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signal generating means coupled to a device for generating a device control signal; and 
control means for: 

( 1 ) receiving a movement related signal and moving at least part of a cursor only 
within the polygon in response to the received movement related signal; and 

(2) in response to a selection event, generating a device control signal corresponding 
to the device control function associated with the one of the plurality of 
selectable regions intersected by the cursor. 

127. The device controller of claim 126 wherein the device includes any one of a wheelchair, a 
household appliance, an appliance for use in an office, a workstation, a robot, and a computer 
peripheral. 

128. For use with a surface comprising a display area, a method of selecting a menu option from a 
plurality of menu options, said method comprising the steps of: 

at least partially delimiting a plurality of selectable regions, each of the selectable 
regions associated respectively with a menu option and each of the selectable regions 
including an invisible subregion outside the display area and a visible subregion on the 
display area, the plurality of visible subregions together at least partially 
circumscribing a region on the display area; 

receiving a movement related signal indicating successive locations with respect to the 
display area: and 

selecting, in response to a dwell event associated with one of the selectable regions, the 
menu option associated with the selectable region associated with the dwell event. 



129. For use with a human interface system wherein a body member of an operator may indicate 
successive locations on a surface, the surface including a display area, the display area 
having thereon a plurality of selectable regions, each of the selectable regions associated 
respectively with a menu option and the plurality of selectable regions together at least 
partially circumscribing a region on the display area, a method of selecting a menu option 
from a plurality of menu options, said method comprising the steps of: 
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confining each of the successive locations to the display area; and 

selecting, in response to a dwell event associated with one of the selectable regions, the 
menu option associated with the selectable region associated with the dwell event. 

130. A method of speaking using a speech synthesis system including a display and a speech 
synthesizer, said method comprising the steps of: 

displaying a plurality of selectable regions within a polygon on the display, each 
selectable region adjacent a side of the polygon and one or more of the selectable 
regions associated respectively with a sequence of one or more characters, the plurality 
of selectable regions together at least partially circumscribing a region on the display; 

receiving a movement related signal and moving at least pan of a cursor only within 
the polygon responsive to the movement related signal; 

repetitively; 

(i) in response to a first quantity equalling or exceeding a predetermined quantity, 
the first quantity being a function of the durations of one or more successive 
periods of intersection of the cursor and one of the one or more selectable 
regions, selecting the sequence associated with the intersected selectable region: 
and 

(ii) appending the selected sequence to at least one previously selected sequence: 
and 

speaking, by means of the speech synthesizer, the word spelled by the appended 
sequences. 

131. A method of inputting data to a computer program for an operator having impaired motor 
capability, said method comprising the steps of: 

displaying a plurality of selectable regions within a polygon on a display, each 
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selectable region adjacent a side of the polygon and each selectable region associated 
respectively with an input for the computer program, the plurality of selectable regions 
together at least partially circumscribing a region on the display; 

receiving a movement related signal and moving at least part of a cursor only within 
the polygon responsive to the movement related signal: and 

in response to a first quantity equalling or exceeding a predetermined quantity, the first 
quantity being a function of the durations of one or more successive periods of 
intersection of the cursor and one of the selectable regions, inputting the input 
associated with the intersected selectable region to the computer program. 

132. An apparatus for indicating dwell time comprising: 

(a) a surface with respect to which a region may be at least partially delimited; 

(b) movement receiving means for receiving a movement related signal indicating a first 
location intersecting the region and, at a later time, a second location intersecting the 
region; and 

(c) indicating means for indicating at least the difference between the time the second 
location occurs and the time the first location occurs. 

133. The apparatus of claim 132 wherein the indicating means includes means for determining the 
difference between the time the second location occurs and the time the first location occurs. 

134. The apparatus of claim 132 wherein the indicating means includes an indicator and means for 
modifying the appearance of the indicator. 

135. The apparatus of claim 132 wherein the region intersects the surface. 

136. The apparatus of claim 1 32 wherein the region is adjacent the surface. 

137. The apparatus of claim 132 wherein the movement related signal further indicates, at a time 
later than the time the second location occurs, a third location not intersecting the region and 
the indicating means is further operative to indicate the non-intersection of the third location 
and the region. 
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1 38. The apparatus of claim 132 wherein the movement related signal further indicates, at a time 
later than the time the second location occurs, a third location not intersecting the region and, 
at a later time, indicates a fourth location not intersecting the region and the indicating means 

5 is further operative to indicate at least the difference between the time the fourth location 

occurs and the time the third location occurs. 

1 39. The apparatus of claim 1 32 wherein the indicating means includes means for producing an 
output signal which varies in at least one way as the difference between the time the second 

10 location occurs and the time the first location occurs increases and varies in at least the 

opposite way as the difference between the time the fourth location occurs and the time the 
third location occurs increases. 

140. An apparatus for indicating dwell time comprising: 
15 (a) a pointer; 

(b) a detector; and 

(c) an indicator for indicating at least the duration of a period the pointer indicates the 
detector. 

20 141. The apparatus of claim 1 40 wherein the indication of the duration of the period of detection 
includes a visible signal. 

142. The apparatus of claim 140 wherein the indication of the duration of the period of detection 
includes an audible signal. 

25 

143. The apparatus of claim 140 wherein the indication of the duration of the period of detection 
includes a tactile signal. 

144. An apparatus for indicating the remaining dwell time required to select a selectable region, 
30 said apparatus comprising: 

(a) a surface with respect to which the selectable region may be at least partially 
delimited: 

(b) movement receiving means for receiving a movement related signal indicating 
successive locations with respect to the surface; and 
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(c) indicating means for modifying in a first manner the appearance of a first indicating 
region on the surface responsive to a period of intersection of two or more of the 
successive locations and the selectable region not equalling or exceeding a 
predetermined period and for modifying in a second manner the appearance of a 
second indicating region on the surface responsive to a period of intersection of two or 
more of the successive locations and the selectable region equalling or exceeding the 
predetermined period. 

145. The apparatus of claim 144 wherein the modification in the first manner includes a slight 
modification and the modification in the second manner includes a marked modification. 

146. The apparatus of claim 144 wherein the modification in the first manner includes a 
modification in brightness. 

147. The apparatus of claim 144 wherein the modification in the second manner includes a 
modification in hue. 

1 48. The apparatus of claim 1 44 wherein the first indicating region intersects the second 
indicating region. 

1 49. The apparatus of claim 1 44 wherein one of the first and second indicating regions intersects 
the selectable region. 

150. The apparatus of claim 1 44 wherein one of the first and second indicating regions is 
coterminous with the selectable region. 

151. An apparatus for indicating dwell time comprising: 

(a) a surface with respect to which a plurality of regions may be at least partially 
delimited; 

(b) movement receiving means for receiving a movement related signal indicating a first 
location intersecting any one of the plurality of regions and. at a later time, a second 
location intersecting the region intersected by the first location: and 

(c) a plurality of indicators, each associated respectively with a region, for indicating at 
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least the difference between the time the second location occurs and the time the first 
location occurs. 



1 52. The apparatus of claim 1 5 1 wherein each indicator is adjacent its associated region. 

153. The apparatus of claim 151 wherein each indicator intersects its associated region. 

1 54. An apparatus for selecting a menu option from a plurality of menu options, said apparatus 
comprising: 

(a) a surface with respect to which a plurality of selectable regions may be at least 
partially delimited, each selectable region associated respectively with a menu option; 

(b) movement receiving means for receiving a movement related signal indicating 
successive locations with respect to the surface: and 

(c) selection means for selecting, in response to a dynamic dwell event, the menu option 
associated with the selectable region intersected by the location of the successive 
locations whose occurrence triggers the dynamic dwell event. 

155. In a human interface system wherein a body member of an operator may indicate successive 
locations with respect to a surface, a menu option selector comprising: 

(a) the surface with respect to which a selectable region may be at least partially 
delimited, the selectable region associated with a menu option; 

(b) a detector for detecting the successive locations; and 

(c) responsive to: 

( 1 ) the duration of a first period during which two or more of the successive 
locations intersect the selectable region; 

(2) the duration of a second period, occurring after the first period, during which two 
or more of the successive locations do not intersect the selectable region; and 

(3) the duration of a third period, occurring after the second period, during which 
two or more of the successive locations intersect the selectable region; 

a selector for selecting the menu option. 

1 56. A device controller comprising: 

(a) display means for displaying a plurality of selectable regions on a surface, each 
selectable region associated respectively with a device control function: 
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(b) movement receiving means for receiving a movement related signal indicating 
successive locations on the surface; 

(c) indicating means for indicating at least the duration of a period of intersection of two 
or more of the successive locations indicated by the movement related signal and a 

5 selectable region; and 

(d) initiation means, responsive to a dwell event, for initiating the control function 
associated with the intersected selectable region. 

1 57. The device controller of claim 1 56 further comprising a controlled device, responsive to the 
10 initiation means, for performing the device control function. 

1 58. The device controller of claim 1 57 wherein the controlled device includes one of a household 
appliance and a motorized wheelchair. 

15 1 59. A computer access device for an operator having impaired motor control, said device 
comprising: 

(a) a computer system, including a display on which my be displayed a plurality of 
selectable regions, each of the selectable regions associated respectively with a 
sequence of one or more characters, and on which display a moveable cursor may be 

20 displayed and then moved responsive to a movement related signal, the computer 

system being further capable of executing an application program operative to accept 
signals representing any of the plurality of sequences of one or more characters; and 

(b) an access program, for execution on the computer system, for indicating at least the 
duration of a period of intersection of the cursor and any one of the one or more 

25 selectable regions and for inputting to the application program signals representing the 

sequence of one or more characters associated with the intersected selectable region in 
response to the period of intersection of the cursor and the intersected selectable region 
equalling or exceeding a predetermined period. 

30 160. For use with a general purpose computer system including a display with respect to which 
may be at least partially delimited a plurality of selectable regions, each of the selectable 
regions associated respectively with a sequence of one or more characters, wherein the 
general purpose computer system is capable of receiving a movement related signal 
indicating successive locations with respect to the display, and wherein the general purpose 
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computer system is capable of executing an application program; 

(a) a medium readable by the general purpose computer system; and 

(b) a program, stored on the medium and executable by the general purpose computer 
system, for: 

(1 ) indicating at least the duration of a period of intersection of two or more of the 
successive locations and any one of the selectable regions; and 

(2) passing to the application program the sequence of one or more characters 
associated with the intersected selectable region in response to the duration of 
the period of intersection equalling or exceeding a predetermined period. 

161. For use with a processing unit, 

(a) said processing unit coupled to a display on which may be displayed a plurality of 
selectable regions, each region associated respectively with a sequence of one or more 
characters: 

(b) said processing unit being capable of receiving a movement related signal indicating 
successive locations on the display; 

(c) said processing unit being capable of executing an application program; 

an access program, executable on the processing unit, for an operator having impaired motor 
control, said access program operative to indicate at least the duration of a period of 
intersection of two or more of the successive locations and an intersected selectable region 
and to pass to the application program the sequence of one or more characters associated 
with the intersected selectable region in response to the duration of the period of intersection 
equalling or exceeding a predetermined period. 

162. The computer access device of claim 161 wherein the access program and the application 
program include at least some common code. 

163. For use with a computer system, a computer access device for an operator having impaired 
motor control, said device comprising: 

(a) a pointer; 

(b) a detector associated with a sequence of one or more characters; 

(c) an indicator for indicating at least the duration of a period the pointer indicates the 
detector; and 
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(d) a processing unit coupled to the computer system for inputting to the computer 

system, in response to a dwell event, signals representing the sequence of one or more 
characters associated with the detector. 



164. A speech synthesis system comprising: 

(a) a display with respect to which may be delimited a plurality of selectable regions, each 
of the selectable regions associated respectively with a sequence of one or more 
characters: 

(b) a speech synthesizer: and 

(c) control means for 

(1) receiving a movement related signal indicating successive locations with respect 
to the surface; 

(2) repetitively 

(a) indicating the duration of a first period of intersection of two or more of 
the successive locations and a first one of the selectable regions: and 

(b) in response to the duration of a second period of intersection of two or 
more of the successive locations and the first one of the selectable regions 
equalling or exceeding a predetermined period, appending the sequence 
associated with the first one of the selectable regions to at least one 
previously selected sequence; and 

(3) speaking, by means of the speech synthesizer, the word spelled by the appended 
sequences. 



165. A speech synthesis system comprising: 

(a) a display on which may be displayed a plurality of selectable regions, each selectable 
region associated respectively with a sequence of one or more characters; 

(b) a speech synthesizer; and 

(c) control means for: 

( 1 ) receiving a movement related signal indicating: 

(A) a first location intersecting any one of the plurality of selectable regions; 

(B) at a time after the first location occurs, a second location intersecting the 
selectable region intersected by the first location; 

(C) at a time after the second location occurs, a third location not intersecting 
the selectable region intersected by the first location; 



147 



WO 96/30822 



PCTYUS95/03591 



(D) at a time after the third location occurs, a fourth location not intersecting 
the selectable region intersected by the first location: 

(E) at a time after the fourth location occurs, a fifth location intersecting the 
selectable region intersected by the first location: and 

(F) at a time after the fifth location occurs, a sixth location intersecting the 
selectable region intersected by the first location: 

(2) repetitively: 

(A) responsive to the difference between the time the second location occurs 
and the time the first location occurs, increasing the brightness of the 
selectable region; 

(B) responsive to the difference between the time the fourth location occurs 
and the time the third location occurs, decreasing the brightness of the 
selectable region: and 

(C) responsive to a quantity that is a function of: 

(i) the difference between the time the second location occurs and the 
time the first location occurs; 

(ii) the difference between the time the fourth location occurs and the 
time the second location occurs; and 

(iii) the difference between the time the sixth location occurs and the 
time the fifth location occurs; 

equalling or exceeding a predetermined quantity, appending the sequence 
associated with the selectable region to at least one previously selected 
sequence: and 

(3) speaking, by means of the speech synthesizer, the word spelled by the appended 
sequences. 

166. A data entry system comprising: 

(a) a display having thereon a plurality of selectable data entry items and a region for 
indicating the number of units of the data entry item selected; 

(b) a plurality of detectors, each associated respectively with one of the selectable data 
entry items; 

(c) a pointer for emitting signals directed at the data entry items: and 

(d) control means for controlling the operation of said data entry system, the control 
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means including: 

(1 ) first indicating means for producing signals in response to the signals falling on 
the detectors, said first indicating means giving a first indication of the period of 
time that the pointer has been directed to a first one of the data entry items; 
5 (2) second indicating means for producing signals in response to the signals falling 

on the detectors, said second indicating means giving a second indication that 
the pointer has been directed to a second one of the data entry items for at least a 
predetermined period of time and that the second one of the data entry items is 
selected for entry into the data entry system: and 
10 (3) third indicating means for indicating in the region the number of units of the 

data entry item selected. 

167. A order entry system comprising: 

(a) a display having thereon a plurality of selectable order entry items, a first region for 
15 indicating the number of units of the order entry item selected and a second region for 

indicating the total monetary value of the. units selected; 

(b) a plurality of detectors, each associated respectively with one of the selectable order 
entry items; 

(c) a pointer for emitting signals directed at the order entry items; and 

20 (d) control means for controlling the operation of said order entry system, the control 

means including 

( 1 ) first indicating means for producing signals in response to the signals falling on 
the detectors, said first indicating means giving a first indication of the duration 
of the period that the pointer has been directed to a first one of the order entry 

25 items; 

(2) second indicating means for producing signals in response to the signals falling 
on the detectors, said second indicating means giving a second indication that 
the pointer has been directed to a second one of the order entry items for at least 
a predetermined period of time and that the second one of the order entry items 

30 is selected for entry into the order entry system; 

(3) third indicating means for indicating in the first region the number of units of the 
order entry item selected; 

(4) totaling means for totaling the monetary value of the units selected: and 

(5) fourth indicating means for indicating in the second region the total monetary 
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value of the units selected. 

168. The order entry system of claim 167 wherein the first indicating means further includes third 
indicating means for giving a third indication of the duration of the period that the pointer is 
not directed to an order entry item to which it was previously directed. 

169. The order entry system of claim 167 wherein the display further includes a pictorial 
representation for at least some of the order entry items and wherein the detectors are located 
within each pictorial representation. 

1 70. The order entry system of claim 1 69 wherein the pictorial representation is in the form of a 
food item at a fast- food restaurant. 

171. The order entry system of claim 167 wherein the pointer includes: 

(a) an infrared light source; and 

(b) a lens for producing a narrow beam of light in cooperation with the infrared light 
source. 

1 72. The order entry system of claim 1 67 further comprising a stationary housing for housing the 
pointer, said housing positioned relative to the display to enable a person using the order 
entry system and seated in a vehicle to pick up the pointer, use it, and return it to the 
stationary housing while seated in the vehicle. 

1 73. An apparatus for selecting a menu option from a plurality of menu options, said apparatus 
comprising: 

(a) a surface including a plurality of detectors each associated respectively with a menu 
option; 

(b) a pointer operative to emit energy detectable by each of the plurality of detectors; and 

(c) responsive to: 

( 1 ) the duration of a first period during which a particular detector detects energy 
emitted by the pointer above a first threshold; 

(2) the duration of a second period, occurring after the first period, during which the 
particular detector does not detect energy emitted by the pointer above a second 
threshold: and 
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(3) the duration of a third period, occurring after the second period, during which the 
particular detector detects energy emitted by the pointer above a third threshold; 
a selector for selecting the menu option associated with the particular detector. 

1 74. The apparatus of claim 1 73 wherein the selector selects the menu option associated with the 
particular detector in response to the sum of a multiple of the duration of the first period and 
a multiple of the duration of the third period minus a multiple of the duration of the second 
period equalling or exceeding a predetermined quantity. 

175. An apparatus for inputting words, said apparatus comprising. 

(a) a display with respect to which may be delimited a selectable region associated with a 
word; 

(b) word receiving means for receiving a signal representing the word; and 

(c) control means for: 

( 1 ) receiving a movement related signal indicating successive locations with respect 
to the surface; 

(2) indicating the duration of a period of intersection of two or more of the 
successive locations and the selectable region; and 

(3) inputting, in response to the period of intersection equalling or exceeding a 
predetermined period, the signal representing the word to the word receiving 
means. 

1 76. The apparatus of claim 175 wherein the word belongs to an ideographic language. 

1 77. A method of indicating the remaining dwell time required to select a selectable region, said 
method comprising the steps of: 

displaying the selectable region on a surface; 

receiving a movement related signal indicating a first location intersecting the selectable 
region and, at a time after the first location occurs, a second location intersecting the 
selectable region; and 

indicating the difference between the time the second location occurs and the time the first 



151 



WO 96/30822 



PCT/US95/03591 



location occurs. 

178. The method of claim 1 77 further comprising the steps of: 

5 receiving a movement related signal indicating, at a time after the time the second location 

occurs, a third location not intersecting the selectable region; and 

indicating the non-intersection of the third location and the selectable region. 

10 179. The method of claim 1 78 further comprising the steps of: 

receiving a movement related signal indicating, at a time after the time the third location 
occurs, a fourth location not intersecting the selectable region; and 

15 indicating the difference between the time the fourth location occurs and the time the third 

location occurs. 

1 80. In a human interface system including a surface on which may be displayed a cursor whose 
movement is responsive to a first movement related signal, and with respect to which surface 
20 may be at least partially delimited a plurality of selectable regions, each of the plurality of 

selectable regions associated respectively with a menu option, an apparatus for selecting any 
one of the plurality of menu options, said apparatus comprising: 

(a) movement related signal receiving means for receiving the first movement related 
signal; and 

25 (b) selection means for selecting the menu option associated with the particular one of the 

plurality of selectable regions most nearly along a first cursor path indicated by a first 
cursor location and the first movement related signal in advance of an intersection of 
the cursor and the particular selectable region. 

30 181. The apparatus of claim 1 80 wherein the movement related signal receiving means is further 
operative to receive a second movement related signal occurring subsequent to the first 
movement related signal; wherein the selection means is further operative to select the menu 
option associated with the specific one of the plurality of selectable regions most nearly 
along a second cursor path indicated by a second cursor location and the second movement 
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related signal in advance of an intersection of the cursor and the specific selectable region; 
wherein the first and second cursor paths are parallel; and wherein the particular selectable 
region is not the specific selectable region. 

182. In a human interface system including a surface on which may be displayed a cursor whose 
movement is responsive to a first movement related signal, and with respect to which surface 
may be at least partially delimited a plurality of selectable regions, each of the plurality of 
selectable regions associated respectively with a menu option, an apparatus for selecting any 
one of the plurality of menu options, said apparatus comprising: 

(a) movement related signal receiving means for receiving the first movement related 
signal; and 

(b) selection means for selecting the menu option associated with the particular one of the 
plurality of selectable regions most nearly along a first cursor path in response to an 
intersection of the cursor and said particular selectable region. 

183. The apparatus of claim 180 or 182 further comprising means for indicating which one of the 
plurality of selectable regions is most nearly along the first cursor path. 

184. The apparatus of claim 180 or 1 82 wherein the surface includes a display area and each of 
the plurality of selectable regions is adjacent an edge of the display area. 

1 85. The apparatus of claim 1 80 or 1 82 wherein the plurality of selectable regions together at least 
partially circumscribes an area on the surface. 

0 

186. An apparatus for selecting an option from a menu, said apparatus comprising: 

(a) means for at least partially delimiting a plurality of selectable regions with respect to a 
surface, each of the plurality of selectable regions associated respectively with a menu 
option; 

(b) movement related signal receiving means for receiving a movement related signal; and 

(c) control means for: 

( 1 ) moving a cursor on the surface in response to the received movement related 
signal; 

(2) changing a selection threshold period responsive to a cursor path: 

(3) detecting an intersection of the cursor and any one of the plurality of selectable 
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regions; and 

(4) in response to the period of the intersection equalling or exceeding the selection 
threshold period, selecting the menu option associated with the intersected 
selectable region. 

5 

187. The apparatus of claim 1 86 wherein the control means is further operative to indicate the 
changed length of the selection threshold period. 

1 88. The apparatus of claim 1 86 wherein one of the plurality of selectable regions has previously 
10 been selected and the amount of change to the selection threshold period varies in response 

to which one of the selectable regions has been previously selected. 

1 89. The apparatus of claim 1 86 wherein the changed selection threshold period equals or exceeds 
a predetermined period. 

15 

190. The apparatus of claim 1 86 wherein one of the plurality of selectable regions has previously 
been selected and each location of the cursor in the cursor path is outside the previously 
selected selectable region. 

20 191. The apparatus of claim 1 90 wherein each location of the cursor in the cursor path is at least a 
predetermined distance from the previously selected selectable region. 

1 92. The apparatus of claim 1 90 wherein each location of the cursor in the cursor path occurs at 
least a predetermined period of time after the selection of the previously selected selectable 
25 region. 



193 . The apparatus of claim 1 86 wherein the amount of change to the selection threshold period 
varies in response to the distance between the intersected selectable region and the location 
of the cursor. 

30 

194. The apparatus of claim 1 86 wherein the amount of change to the selection threshold period 
varies in response to the difference between the time the intersection occurs and the time the 
location of the cursor occurs. 
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195. The apparatus of claim 186 wherein each of the plurality of selectable regions includes a 
subregion visible on the surface and an invisible subregion located outside the surface and 
wherein the plurality of visible subregions together at least partially circumscribes an area on 
the surface. 

196. The apparatus of claim 1 86 wherein the movement related signal is responsive to the 
movement of a body member of an operator having impaired ability to sense the position of 
the body member and the apparatus further comprises tactile indication means for indicating 
with the body member tactilely to the operator the position of the body member. 

197. The apparatus of claim 1 86 wherein one or more of the plurality of menu options represents 
any one of: 

(a) a prefix; 

(b an infix: and 

(c) a suffix; 

to be added to a root word. 



198. An apparatus for selecting a menu option from a plurality of menu options, said apparatus 
comprising: 

(a) means for at least partially delimiting a plurality of selectable regions with respect to a 
surface, each of the plurality of selectable regions associated respectively with a menu 
option; 

(b) movement related signal receiving means for receiving a movement related signal 
indicating successive locations on the surface; and 

(c) control means for: 

( 1 ) moving a cursor in a first manner on the surface in response to the movement 
related signal and thereafter detecting an intersection of the cursor and any one 
of the plurality of selectable regions; 

(2) responsive to a first cursor path, moving the cursor in a second manner on the 
surface in response to the movement related signal, wherein the movement of the 
cursor in the second manner is attenuated relative to the movement of the cursor 
in the first manner; and 

(3) in response to the period of the intersection equalling or exceeding a selection 
threshold period, selecting the menu option associated with the intersected 
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selectable region. 

199. The apparatus of claim 1 98 wherein the moving of the cursor in the second manner is further 
responsive to the intersection. 

200. The apparatus of claim 199 wherein the moving of the cursor in the second manner is further 
responsive to a path indicated by successive locations occurring after the intersection. 

201. The apparatus of claim 199 wherein the moving of the cursor in the second manner is further 
responsive to an angle of attack of the cursor path to the intersected selectable region. 

202. The apparatus of claim 198 wherein the plurality of selectable regions together at least 
partially circumscribes an area on the surface. 

203. The apparatus of claim 198 wherein the moving of the cursor in the second manner is further 
responsive to the velocity of the first cursor path. 

204. The apparatus of claim 198 wherein the moving of the cursor in the second manner is further 
responsive to the acceleration of the first cursor path. 

205. The apparatus of claim 198 wherein the moving of the cursor in the second manner is further 
responsive to the proximity of a cursor location in the first cursor path to the selectable 
region most nearly along the first cursor path. 

206. In a human interface system including a first surface on which may be displayed a plurality 
of selectable regions each associated respectively with a menu option, and wherein a body 
member of an operator may indicate successive locations on a second surface, each location 
corresponding to a location on the first surface, a selection controller comprising: 

(a) a direction detector for determining a move direction of the body member responsive 
to two or more of the successive locations on the second surface; and 

(b) a selector for selecting the menu option associated with the particular one of the 
plurality of selectable regions most nearly along a path on the first surface 
corresponding to the move direction of the body member, the selection occurring in 
advance of an intersection of any one of the locations on the first surface and the 



156 



WO 96/30822 



PCT/US95/03591 



particular selectable region. 

207. In a human interface system including a first surface on which may be displayed a plurality 
of selectable regions each associated respectively with a menu option, and wherein a body 
5 member of an operator may indicate successive locations on a second surface, each location 

corresponding to a location on the first surface, a selection controller comprising: 

(a) a direction detector for determining a move direction of the body member responsive 
to tw o or more of the successive locations on the second surface; and 

(b) a selector for selecting the menu option associated with the particular one of the 
10 plurality of selectable regions most nearly along a path on the first surface 

corresponding to the move direction of the body member, the selector being responsive 
to an intersection of one of the locations on the first surface and the particular 
selectable region. 

15 208. In a human interface system including 

a first surface on which may be displayed a plurality of regions each associated 
respectively with a menu option; 

wherein a body member of an operator may indicate successive locations on a second 
surface, each location corresponding to a location on the first surface; and 
20 wherein a particular one of the regions may be selected responsive to a period of 

intersection of two or more of the locations on the first surface and the particular 
region equalling or exceeding a selection threshold period; 
a selection threshold controller comprising: 

(a) a direction detector for determining a move direction of the body member responsive 
25 to two or more of the locations on the second surface; and 

(b) means for changing the selection threshold period responsive to the move direction of 
the body member. 

209. The selection threshold controller of claim 208 further comprising a pointer for indicating 
30 with the body member successive locations on the second surface. 

210. The selection threshold controller of claim 208 wherein the first surface includes the second 
surface. 
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211. The selection threshold controller of claim 208 wherein the second surface includes the first 
surface. 

212. The selection threshold controller of claim 208 wherein the body member of the operator 
includes any one of the operator's shoulder, the operator's arm, the operator's elbow, the 
operator's wrist, the operator's hand, the operator's finger, the operator's thumb, the operator's 
leg, the operator's knee, the operator's ankle, the operator's foot, the operator's toe. the 
operator's hip, the operator's trunk, the operator's neck, the operator's tongue, the operator's 
lip. the operator's eye and the operator's head. 

213. The selection threshold controller of claim 208 wherein the means for changing of the 
selection threshold period includes means for decreasing the selection threshold period 
responsive to the move direction indicating with the body member a path toward the 
particular region. 

214. The selection threshold controller of claim 208 wherein the means for changing of the 
selection threshold period includes means for increasing the selection threshold period 
responsive to the move direction indicating with the body member a path away from the 
particular region. 

215. In a human interface system including 

a first surface on which may be displayed a plurality of regions each associated 
respectively with a menu option: 

wherein a body member of an operator may indicate successive locations on a second 
surface, each location corresponding to a location on the first surface; and 
wherein a particular one of the regions may be selected responsive to a period of 
intersection of two or more of the locations on the first surface and the particular 
region equalling or exceeding a selection threshold period; 

a selection threshold controller comprising a threshold controller for changing the selection 

threshold period responsive to one of: 

( 1 ) a path indicated by two or more of the successive locations on the first surface; and 

(2) one of the locations on the first surface intersecting a predetermined region on the first 
surface. 
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216. An apparatus for selecting an option from a menu, said apparatus for use with a general 
purpose computer system including a display on which may be displayed a cursor whose 
movement is responsive to a movement related signal, said apparatus comprising: 
(a) a medium readable by the general purpose computer system; and 
5 (b) an access program, stored on the medium and executable by the general purpose 

computer system, for: 

(1 ) at least partially delimiting a plurality of selectable regions with respect to the 
display, each of the plurality of selectable regions associated respectively with a 
menu option: and 

10 (2) selecting the menu option associated with the particular one of the plurality of 

selectable regions most nearly along a path indicated by a cursor location on the 
display and the movement related signal in advance of an intersection of the 
cursor and said particular selectable region. 

15 217. An apparatus for selecting an option from a menu, said apparatus for use with a general 
purpose computer system including a display on which may be displayed a cursor whose 
movement is responsive to a movement related signal, said apparatus comprising: 

(a) a medium readable by the general purpose computer system; and 

(b) an access program, stored on the medium and executable by the general purpose 
20 computer system, for: 

( 1 ) at least partially delimiting a plurality of selectable regions with respect to the 
display, each of the plurality of selectable regions associated respectively with a 
menu option: and 

(2) selecting the menu option associated with the particular one of the plurality of 
25 selectable regions most nearly along a path indicated by a cursor location on the 

display and the movement related signal in response to an intersection of the 
cursor and said particular selectable region. 

218. An apparatus for selecting an option from a menu, said apparatus for use with a general 
30 purpose computer system including a display on which may be displayed a cursor whose 

movement is responsive to a movement related signal, said apparatus comprising: 

(a) a medium readable by the general purpose computer system; and 

(b) an access program, stored on the medium and executable by the general purpose 
computer system, for: 



159 



WO 96/30822 



PCT/US95/03591 



(1) at least partially delimiting a plurality of selectable regions with respect to the 
display, each of the plurality of selectable regions associated respectively with a 
menu option; and 

(2) changing a selection threshold period responsive to a cursor path; 

5 (3) detecting an intersection of the cursor and any of the plurality of selectable 

regions; and 

(4) in response to the period of the intersection equalling or exceeding the selection 
threshold period, selecting the menu option associated with the intersected 
selectable region. 

10 

219. An apparatus including a surface, for measuring an individual's ability to move a position 
indicator on the surface in a direct path from a starting location on the surface to an ending 
location on the surface, said position indicator indicative of a position of one of the 
individual's body members, said apparatus comprising: 

15 (a) the surface; 

(b) position indicating with the body member means for indicating with the body member 
on the surface a position of the body member; 

(c) collection means for collecting data indicative of a succession of the indicated 
positions on the surface; and 

20 (d) measurement means for measuring the deviation of the path indicated by the collected 

data from a predetermined path between the starting location and the ending location. 

220. An interactive terminal, including a display on which may be displayed: 

a plurality of selectable regions, each of the plurality of selectable regions associated 
25 respectively with a menu option and a selection threshold period; and 

a cursor whose movement is responsive to the movement of a body member of an 
operator; 
said terminal comprising: 

(a) means for displaying the plurality of selectable regions on the display; and 
30 (b) control means for: 

( 1 ) in response to cursor movement toward a particular one of the plurality of 
selectable regions, decreasing the selection threshold period associated with the 
particular selectable region; 

(2) in response to cursor movement diverging from a path toward the particular 
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selectable region, increasing the selection threshold period associated with the 
particular selectable region; 

(3) detecting an intersection of the cursor and any of the plurality of selectable 
regions: and 

(4) in response to the period of the intersection being equal to or greater than the 
selection threshold period associated with the intersected selectable region, 
selecting the menu option associated with the intersected selectable region. 

221 . The terminal of claim 220 wherein the plurality of selectable regions together at least 
partially circumscribes an area on the display. 

222. A data entry system comprising: 

(a) a computer system including: 

( 1 ) a display on which a moveable cursor may be displayed and then moved 
responsive to a movement related signal and on which a plurality of selectable 
regions may be displayed, each selectable region associated respectively with a 
sequence of one or more characters and a character sequence signal representing 
the associated sequence of one or more characters; and 

(2) a processing unit coupled to the display and capable of executing an application 
program operative to accept any of the character sequence signals; and 

(b) an input program, for execution on the processing unit, for inputting to the application 
program the character sequence signal representing the sequence of one or more 
characters associated with the particular one of the plurality of selectable regions most 
nearly along the path indicated by a cursor location and the movement related signal in 
advance of an intersection of the cursor and said particular selectable region. 

223. A data entry system comprising: 
(a) a computer system including: 

( 1 ) a display on which a moveable cursor may be displayed and then moved 
responsive to a movement related signal and on which a plurality of selectable 
regions may be displayed, each selectable region associated respectively with a 
sequence of one or more characters; and 

(2) a processing unit coupled to the display and capable of executing an application 
program operative to accept any of the plurality of sequences of one or more 
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characters; and 

(b) an input program, for execution on the processing unit, for inputting to the application 
program the sequence of one or more characters associated with the particular one of 
the plurality of selectable regions most nearly along a cursor path, said input to the 
5 application program being responsive to an intersection of the cursor and said 

particular selectable region. 



224. The data entry system of claim 220 wherein the input program further comprises means for 
changing a selection threshold period associated with said particular selectable region and 

10 wherein said input to the application program is further responsive to the duration of the 

period of the intersection equalling or exceeding the selection threshold period. 

225. In a computer system including a display on which may be displayed a moveable cursor, and 
on which may be displayed a menu having a plurality of menu options, each of the menu 

15 options associated respectively with a selectable region on the display, an apparatus for 

selecting any one of the plurality of menu options, said apparatus comprising: 

(a) identification means for identifying which one of the plurality of selectable regions is 
most nearly along a cursor path; and 

(b) selection means for selecting the menu option associated with the identified selectable 
20 region in advance of an intersection of the cursor and the identified selectable region. 



226. In a computer system including a display on which may be displayed a moveable cursor, and 
on which may be displayed a menu having a plurality of menu options, each of the menu 
options associated respectively with a selectable region on the display, an apparatus for 

25 selecting any one of the plurality of menu options, said apparatus comprising: 

(a) means for detecting an intersection of the cursor and any one of the selectable regions; 
and 

(b) selection means, responsive to the detected intersection, for selecting the menu option 
associated with the particular selectable region most nearly along a cursor path. 

30 

227. The apparatus of claim 226 wherein the selection means further comprises means for 
decreasing a selection threshold period associated with the particular selectable region and 
wherein the selection means is further responsive to the duration of the period of the 
intersection equalling or exceeding the selection threshold period. 
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228. A speech synthesis system comprising: 

(a) a display on which may be displayed a plurality of selectable regions, each of the 

selectable regions associated respectively with a sequence of one or more characters; 
5 (b) a speech synthesizer; and 

(c) control means for: 

( 1 ) receiving a movement related signal and moving a cursor on the display 
responsive to the movement related signal; 

(2) repetitively 

10 (A) detecting an intersection of the cursor and any of the plurality of selectable 

regions; 

(B) changing a selection threshold period responsive to a cursor path to the 
intersected selectable region; and 

(C) in response to the period of the intersection equalling or exceeding the 
15 selection threshold period, appending the sequence associated with the 

intersected selectable region to at least one previously selected sequence; 
and 

(3) speaking, by means of the speech synthesizer, the word spelled by the appended 
sequences. 

20 

229. A device controller comprising: 

(a) means for displaying a plurality of selectable regions on a surface, each of the plurality 
of selectable regions associated respectively with a device control function; 

(b) movement related signal receiving means for receiving a movement related signal; 

25 (c) signal generating means coupled to a device for generating a device control signal; and 

(d) control means for: 

(1 ) moving a cursor on the surface in response to the received movement related 
signal; 

(2) changing a selection threshold period responsive to a cursor path; and 

30 (3) detecting an intersection of the cursor and any one of the selectable regions; 

(4) in response to the period of the intersection equalling or exceeding the selection 
threshold period, generating a device control signal corresponding to the device 
control function associated with the intersected selectable region. 
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23 1 . In a computer system, an apparatus for measuring an individual's ability to move one of the 
individual's body members directly from a starting position to an ending position, said 

5 apparatus comprising: 

(a) receiving means, coupled to the computer system, for receiving a sampling among data 
indicative of successive positions of the body member: and 

(b) measurement means, executable on the computer system, for measuring deviation of 
the path indicated by the received position data from a direct path between the starting 

10 position and the ending position. 

232. The apparatus of claim 23 1 further comprising: 

(c) a display, coupled to the computer system, on which a moveable cursor may be 
displayed: and 

15 (d) cursor movement means for moving the cursor on the display responsive to the 

received position data. 

233. An apparatus for selecting an option from a menu, said apparatus comprising: 

(a) means for displaying a plurality of selectable regions on a surface, each of the plurality 
20 of selectable regions associated respectively with a menu option and a selection 

threshold period; and 

(b) movement related signal receiving means for receiving a movement related signal; and 

(c) control means for: 

( 1 ) moving a cursor on the surface in response to the received movement related 
25 signal; 

(2) in response to a cursor path not toward any one of the plurality of selectable 
regions, increasing the selection threshold period associated with said one of the 
plurality of selectable regions; 

(3) detecting an intersection of the cursor and any one of the plurality of selectable 
30 regions; and 

(4) in response to the period of the intersection being equal to or greater than the 
selection threshold period associated with the intersected selectable region, 
selecting the menu option associated with the intersected selectable region. 
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234. A method of selecting an option from a menu, said method comprising the steps of: 



displaying a plurality of selectable regions on a surface, each of the plurality of 
selectable regions associated respectively with a menu option; 

detecting a movement related signal and in response moving a cursor on the surface; 

in response to an intersection of the cursor and the selectable region most nearly along 
a cursor path, selecting the menu option associated with the intersected selectable 
region. 

235. The method of claim 234 further comprising the step of changing a selection threshold period 
associated with the selectable region most nearly along the cursor path and wherein the 
selecting step is further responsive to the duration of the period of the intersection equalling 
or exceeding the selection threshold period. 

236. A method of selecting an option from a plurality of options, each of the plurality of options 
shown adjacent an edge of a display, said method comprising the steps of: 

detecting a movement related signal and in response moving a cursor on the display; 

selecting the option shown most nearly along a cursor path in advance of an 
intersection of the cursor and the selected option; 

237. A method of measuring an individual's ability to move one of the individual's body members 
in a direct path from a starting position to an ending position, said method comprising the 
steps of: 

collecting a sampling among data indicative of successive positions of the body 
member; and 

measuring the deviation of the path indicated by the collected data from a direct path 
between the starting position and the ending position. 
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238. An apparatus for selecting an option from a menu, said apparatus comprising. 

(a) cursor movement means for receiving a movement related signal and for moving a 
cursor on a display responsive to the received movement signal; 

(b) delimit means for delimiting with respect to the display a first plurality of regions and 
a second plurality of selectable regions, each of the first plurality of regions associated 
respectively with one of the second plurality of selectable regions, each of the second 
plurality of selectable regions associated respectively with a menu option, wherein at 
least one of the first plurality of regions is not coterminous with its associated one of 
the second plurality of selectable regions; and 

(c) selection means, responsive to an intersection of the cursor and any one of the first 
plurality of regions and to a selection event associated with the one of the second 
plurality of selectable region associated with the intersected one of the first plurality of 
regions, for selecting the menu option associated with the selectable region associated 
with the selection event. 

239. The apparatus of claim 238 wherein the delimit means includes means for displaying at least 
part of at least one of the first plurality of regions on the display. 

240. The apparatus of claim 238 wherein the delimit means includes means for displaying at least 
part of at least one of the second plurality of selectable regions on the display. 

24 1 . The apparatus of claim 238 wherein each of the at least one of the first plurality of regions 
intersects its associated one of the second plurality of selectable regions. 

242. The apparatus of claim 241 wherein each of the at least one of the first plurality of regions 
shares at least one side with its associated selectable region. 

243 . The apparatus of claim 238 wherein none of the first plurality of regions is coterminous with 
its associated selectable region. 

244. The apparatus of claim 238 wherein the first plurality of regions together at least partially 
circumscribes a first region on the display. 

245. The apparatus of claim 244 wherein the second plurality of selectable regions together at 
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least partially circumscribes a second region on the display. 

246. The apparatus of claim 238 wherein the selection event includes the durations of one or more 
successive periods of intersection of the cursor and the selectable region associated with the 

5 selection event equalling or exceeding a selection threshold period. 

247. The apparatus of claim 238 wherein the selection means includes mean? for receiving a 
switch operation signal: and wherein the first selection event includes: 

( 1 ) an intersection of the cursor and the selectable region associated with the selection 
1 0 event; and 

(2) at or near the time the intersection occurs, receipt of the switch operation signal. 

248. The apparatus of claim 238 wherein the intersected one of the first plurality of regions is 
adjacent another one of the first plurality of regions and the intersection is the earliest 

15 intersection of a plurality of intersections of the cursor and the union of the intersected one of 

the first plurality of regions and the adjacent one of the first plurality of regions. 

249. An apparatus for selecting an option from a menu, said apparatus for use with a general 
purpose computer system including a display on which may be displayed a cursor whose 

20 movement is responsive to a movement related signal, said apparatus comprising: 

(a) a medium readable by the general purpose computer system; and 

(b) an access program, stored on the medium and executable by the general purpose 
computer system, for: 

(1) delimiting a first plurality of regions and a second plurality of selectable regions 
25 with respect the display, each of the first plurality of regions associated 

respectively with one of the second plurality of selectable regions, each of the 
second plurality of selectable regions associated respectively with a menu 
option, the second plurality of selectable regions together at least partially 
circumscribing a third region on the display; wherein at least one of the first 
30 plurality of regions is not coterminous with its associated one of the second 

plurality of selectable regions; and 

(2) responsive to an intersection of the cursor and any one of the first plurality of 
regions and to a selection event associated with the one of the second plurality of 
selectable region associated with the intersected one of the first plurality of 
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regions, selecting the menu option associated with the selectable region 
associated with the selection event. 

250. For use with a computer system including a display, the computer system being capable of 

5 executing an application program, a computer access device for an operator having impaired 

motor control, said device comprising: 

(a) cursor movement means for receiving a movement related signal and for moving the 
cursor on the display responsive to the received movement signal: 

(b) display means for displaying a first plurality of regions on the display, each of the first 
10 plurality of regions associated respectively with one of a second plurality of selectable 

regions, each of the first plurality of regions including all the area of its associated one 
of the second plurality of selectable regions, the first plurality of regions together at 
least partially circumscribing a third region on the display on which may be displayed 
at least part of the output of the application program, the second plurality of selectable 

15 regions together at least partially circumscribing the third region, each of the second 

plurality of selectable regions associated respectively with a menu option, at least one 
of the menu options representing an input for the application program; and wherein the 
display means, responsive to an intersection of the cursor and any one of the first 
plurality of regions, is operative to display on the display at least part of the one of the 

20 second plurality of selectable regions associated with the intersected region; and 

(c) input means, responsive to a dwell event associated with any one of the second 
plurality of selectable regions associated with one of the at least one menu options 
representing an input for the application program, for inputting to the application 
program the menu option associated with the selectable region associated with the 

25 dwell event. 

25 1 . In a human interface system wherein a body member of an operator may indicate successive 
locations on a surface, a menu option selector for selecting a menu option from a plurality of 
menu options, said menu option selector comprising: 

30 (a) the surface on which may be delimited a first plurality of regions and a second 

plurality of selectable regions, each of the first plurality of regions associated 
respectively with one of the second plurality of selectable regions, each of the second 
plurality of selectable regions associated respectively with a menu option: and wherein 
at least one of the first plurality of regions is not coterminous with its associated one of 
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regions, selecting the menu option associated with the selectable region 
associated with the selection event. 

250. For use with a computer system including a display, the computer system being capable of 
executing an application program, a computer access device for an operator having impaired 
motor control, said device comprising: 

(a) cursor movement means for receiving a movement related signal and for moving the 
cursor on the display responsive to the received movement signal; 

(b) display means for displaying a first plurality of regions on the display, each of the first 
plurality of regions associated respectively with one of a second plurality of selectable 
regions, each of the first plurality of regions including all the area of its associated one 
of the second plurality of selectable regions, the first plurality of regions together at 
least partially circumscribing a third region on the display on which may be displayed 
at least part of the output of the application program, the second plurality of selectable 
regions together at least partially circumscribing the third region, each of the second 
plurality of selectable regions associated respectively with a menu option, at least one 
of the menu options representing an input for the application program; and wherein the 
display means, responsive to an intersection of the cursor and any one of the first 
plurality of regions, is operative to display on the display at least part of the one of the 
second plurality of selectable regions associated with the intersected region; and 

(c) input means, responsive to a dwell event associated with any one of the second 
plurality of selectable regions associated with one of the at least one menu options 
representing an input for the application program, for inputting to the application 
program the menu option associated with the selectable region associated with the 
dwell event. 

25 1 . In a human interface system wherein a body member of an operator may indicate successive 
locations on a surface, a menu option selector for selecting a menu option from a plurality of 
menu options, said menu option selector comprising: 

(a) the surface on which may be delimited a first plurality of regions and a second 
plurality of selectable regions, each of the first plurality of regions associated 
respectively with one of the second plurality of selectable regions, each of the second 
plurality of selectable regions associated respectively with a menu option: and wherein 
at least one of the first plurality of regions is not coterminous with its associated one of 
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the second plurality of selectable regions; and 
(b) in response to: 

( 1 ) an intersection of one of the successive locations and any one of the first 
plurality of regions; and 

(2) thereafter, the durations of one or more successive periods of intersection of two 
or more of the successive locations and the one of the second plurality of 
selectable regions associated with the intersected one of the first plurality of 
regions, 

a selector for selecting the menu option associated with the one of the second plurality 
of selectable regions associated with the intersected one of the first plurality of regions. 

252. A speech synthesis system comprising: 

(a) a computer system including a display on which may be displayed a first plurality of 
regions, each of the first plurality of regions associated respectively with one of a 
second plurality of selectable regions, each of the second plurality of selectable regions 
associated respectively with a sequence of one or more characters; 

(b) a speech synthesizer coupled to the computer system; and 

(c) control means for: 

(1) receiving a movement related signal and moving a cursor on the display 
responsive to the movement related signal; 

(2) repetitively, responsive to: 

(A) an intersection the cursor any one of the first plurality of regions; and 

(B) thereafter, the durations of one or more periods of intersection of the 
cursor and the one of the second plurality of selectable regions associated 
with the intersected one of the first plurality of regions. 

appending to at least one previously selected sequence the sequence associated 
with the one of the second plurality of selectable regions associated with the 
intersected one of the first plurality of regions; and 

(3) speaking, by means of the speech synthesizer, the word spelled by the appended 
sequences. 

253. A method of selecting an option from a menu, said method comprising the steps of: 
receiving a movement related signal and moving a cursor on a surface responsive thereto; 
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delimiting a first plurality of regions and a second plurality of selectable regions with respect 
to a surface, each of the first plurality of regions associated respectively with one of the 
second plurality of selectable regions, each of the second plurality of selectable regions 
associated respectively with a menu option: wherein at least one of the first plurality of 
regions is not coterminous with its associated one of the second plurality of selectable 
regions; and 

responsive to an intersection of the cursor and any one of the first plurality of regions and to 
a selection event associated with the one of the second plurality of selectable region 
associated with the intersected one of the first plurality of regions, selecting the menu option 
associated with the selectable region associated with the selection event. 

254. An apparatus, including a surface on which may be displayed a moveable cursor, said 
apparatus allowing an operator to align a pointer with a predetermined location on the 
surface, said apparatus comprising: 

(a) display means for displaying the cursor on the surface; 

(b) movement related signal receiving means for receiving a movement related signal; and 

(c) control means for moving the cursor in response to the movement related signal and 
for thereafter inhibiting movement of the cursor for a first period of time in response to 
an operator action detected by the movement related signal receiving means. 

255. The apparatus of claim 254 wherein the surface includes a display area and the display means 
further includes means for confining at least part of the cursor to the display area. 

256. The apparatus of claim 254 wherein the operator action includes one of: 

( 1 ) an intersection of at least part of the cursor and a region on the surface for a second 
period of time; and 

(2) a switch operation. 

257. The apparatus of claim 254 further comprising start indication means for indicating the start 
of the first period of time and end indication means for indicating the end of the first period 
of time. 

258. The apparatus of claim 257 wherein the start indication means includes changing means for 
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changing the appearance of the cursor and the end indication means includes restoring means 
for restoring the appearance of the cursor to the appearance it had before the changing means 
changed it. 

259. The apparatus of claim 254 wherein the movement related signal is responsive to the 
movement of a body member of an operator having impaired ability to sense the position of 
the body member and the apparatus further comprises tactile indication means for indicating 
tactilely to the operator the position of the body member. 

260. An apparatus, including a surface on which may be displayed a moveable cursor, said 
apparatus allowing an operator to align a pointer producing a movement related signal with 
the cursor, said apparatus comprising: 

(a) display means for displaying the cursor on the surface; 

(b) movement related signal receiving means for receiving the movement related signal: 

(c) detection means for detecting a switch operation; and 

(d) control means for moving the cursor responsive to the movement related signal and for 
thereafter inhibiting movement of the cursor for a first period of time in response to a 
switch operation detected by the detection means. 

261 . An apparatus, including a surface on which may be displayed a moveable cursor, said 
apparatus allowing an operator to align a pointer with a predetermined location on the 
surface, said apparatus comprising: 

(a) display means for displaying the cursor on the surface; 

(b) movement related signal receiving means for receiving a movement related signal; 

(c) switch operation signal receiving means for detecting a switch operation signal; and 

(d) control means for 

( 1 ) moving the cursor responsive to the movement related signal; 

(2) thereafter, in response to a detected first switch operation, inhibiting movement 
of the cursor; and 

(3) thereafter, in response to a detected second switch operation, moving the cursor 
responsive to the movement related signal. 

262. In a human interface system including a surface on which may be displayed a cursor whose 
movement usually tracks the movement of a body member of an operator, an apparatus for 
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aligning the body member of the operator with a predetermined location on the surface, said 
apparatus comprising a cursor controller for: 

(a) detecting movement of the body member of the operator; 

(b) detecting an operator action; and 

5 (c) moving the cursor on the surface in response to the movement of the body member of 

the operator and thereafter inhibiting movement of the cursor for a first period of time 
in response to the detected operator action. 

263. The apparatus of claim 262 wherein the cursor controller is further operative to display a 
10 target at the predetermined location on the surface during at least part of the first period of 

time and to display the cursor at the predetermined location at the end of the first period of 
time. 



264. The apparatus of claim 262 wherein the cursor controller is further operative to display a 
15 target in or in close proximity to the location of the cursor immediately prior to the start of 

the first period of time. 



265 The apparatus of claim 262 wherein the operator action includes one of: 

(1 ) the intersection of the cursor and a region on the display area for a second period of 
20 time; and 

(2) a switch operation. 

266. In an interactive terminal including: 

( 1 ) a display on which a moveable first cursor may be displayed: and 

25 (2) a region on the display, said region not greater in size than the display, 

wherein movement of the hotspot of the first cursor within the region reflects the movement 
of at least one body member of the operator of the interactive terminal and wherein 
movement of a body member that would, if accurately reflected in cursor movement, take the 
cursor out of the region, is not reflected in cursor movement, such that the hotspot of the 

30 cursor remains on the display, said interactive terminal comprising a cursor controller for: 

(a) inhibiting display of the first cursor; 

(b) displaying a stationary target on the display for a prescribed period of time: and then 

(c) displaying a moveable stationary target in or in close proximity to the position of the 
stationary target. 
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so that the operator may align the body member with the position of the stationary target. 

267. For use with a computer system including a display upon which a moveable cursor may be 
displayed, an apparatus for temporarily inhibiting cursor movement, said apparatus 
comprising: 

(a) a pointer; and 

(b) a program executable by the computer system for moving the cursor responsive to the 
pointer and for thereafter inhibiting cursor movement for a prescribed period of time in 
response to an operator action. 

apparatus comprising: 

a computer system including a display on which a moveable cursor may be displayed; 
movement related signal receiving means, coupled to the computer system, for 
receiving a movement related signal: and 

program means, coupled to the computer system, for moving the cursor responsive to 
the movement related signal and for thereafter temporarily inhibiting cursor movement 
in response to a signal detected by the movement related signal receiving means. 

269. An apparatus allowing an operator having impaired motor control to align a pointer with a 
cursor, said apparatus comprising: 

(a) a display area; 

(b) movement related signal receiving means for receiving a movement related signal; 

(c) operator action detection means for detecting an operator action: and 

(d) control means for: 

(1) displaying a first cursor on the display area; 

(2) moving the first cursor responsive to the movement related signal while 
confining the hotspot of the first cursor to the display area; then 

(3) in response to the detected operator action, displaying a second cursor differing 
in configuration from the first cursor at a predetermined location on the display 
area during a first period of time; and 

(4) at the end of the first period of time, displaying the first cursor in or in close 
proximity to the predetermined location on the display area. 

270. The apparatus of claim 269 further comprising means for displaying a plurality of regions on 
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the display area and wherein the predetermined location is a location associated with the 
region intersected by the hotspot of the first cursor when the operator action detection means 
detects the operator action. 

271 . An apparatus allowing an operator having below normal motor capabilities to bring a pointer 
and a cursor coupled to the pointer into alignment with each other, said apparatus 
comprising: 

(a) a display area; 

(b) a processor operative to: 

( 1 ) display a first cursor on the display area, 

(2) move the first cursor responsive to operator movement while confining at least 
part of the first cursor to the display area; and then 

(3) responsive to an intersection of the first cursor and a region on the display area 
for a first period of time: 

(i) remove the first cursor from the display area for a second period of time; 
and 

(ii) during the second period of time, display at a predetermined location on 
the display area a second cursor differing in configuration from the first 
cursor. 

272. For use with a system comprising a pointer and a surface on which is displayed a first cursor 
whose movement on the surface is responsive to the movement of the pointer, a method of 
aligning the first cursor with a predetermined location on the surface, said method 
comprising the steps of: 

(a) producing the first cursor on the surface; 

(b) moving the first cursor responsive to the movement of the pointer, said movement 
restricted to a region on the surface; and then 

(c) responsive to an operator action, producing a second cursor at a predetermined location 
on the surface for a period of time; and 

(d) positioning the first cursor in or in close proximity to the predetermined location.. 

273 . The method of claim 272 further comprising the steps of: 

indicating with the body member the start of the period of time: and 
indicating with the body member the end of the period of time. 
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274 The method of claim 273 wherein the step of indicating with the body member the start of 
the period of time includes generating a first signal and the step of indicating with the body 
member the end of the period of time includes generating a second signal distinct from the 
first signal. 

275. In a menu interface system, an apparatus for ordering menu options comprising: 

(a) order means for ordering a plurality of named menu options in an order responsive to 
the length of the name of each named menu option: and 

(b) display means for displaying the plurality, of named menu options in said order. 

276. The apparatus of claim 275 wherein the displayed plurality of named menu options together 
at least partially circumscribe a region on a display. 

277. The apparatus of claim 275 wherein each of the displayed menu options is adjacent at least 
one other display menu option. 

278. The apparatus of claim 275 wherein the order means includes an arithmetic logic unit. 

279. The apparatus of claim 275 wherein the name of each named menu option includes letters 
and the length of each name includes the number of letters in the name. 

280. The apparatus of claim 275 wherein the name of each named menu option includes letters 
and the order means is further responsive to the alphabetic order of each name. 

28 1 . In a menu interface system, an apparatus for displaying words, said apparatus comprising: 

(a) order means for ordering a plurality of words in an order responsive to the number of 
letters in each word; and 

(b) display means for displaying the plurality of words in said order; 

whereby an operator who knows the relative number of letters in a desired word and one of 
the plurality of words may limit his search for the desired word. 

282. The apparatus of claim 281 wherein the display means includes a display. 
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283. An apparatus for selecting a sequence of one or more words, said apparatus comprising: 

(a) display means for displaying on a display a plurality of sequences of one or more 
words in an order responsive to the length of each sequence; 

(b) cursor movement means for receiving a movement related signal and for moving a 
cursor on the display responsive to the received movement signal; and 

(c) selection means, responsive to a selection event associated with any one of the 
sequences, for selecting the sequence associated with the selection event. 

284. The apparatus of claim 283 wherein each sequence is composed of one word. 

285. The apparatus of claim 283 wherein each sequence includes one or more characters and the 
length of each sequence includes the number of characters in the sequence. 

286. The apparatus of claim 283 wherein the selection event includes an intersection of the cursor 
and a region on the display associated with the word associated with the selection event. 

287. The apparatus of claim 283 wherein at least one of the sequence represents a sentence. 

288. The apparatus of claim 283 further comprising means for modifying the selected sequence. 

289. A speech synthesis system comprising: 

(a) a display for displaying a plurality of named menu options in an order responsive to 
the length of the name of each named menu option, each of the named menu options 
associated respectively with a sequence of one or more characters; 

(b) a speech synthesizer; and 

(c) control means for: 

( 1 ) selecting, responsive to a selection event associated with any one of the named 
menu options, the sequence of one or more characters associated with the named 
menu option associated with the selection event; and 

(2) speaking, by means of the speech synthesizer, the selected sequence of one or 
more characters. 

290. For use with a computer capable of executing an application program, a method of data entry 
comprising the steps of: 
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(a) displaying a plurality of named menu options in an order responsive to the length of 
the name of each named menu option, each of the named menu options associated 
respectively with a sequence of one or more characters; and 

(b) inputting to the application program, responsive to a selection event associated with 
one of the named menu options, the sequence of one or more characters associated 
with the named menu option associated with the selection event. 

291 . The method of claim 290 wherein the name of at least one of the named menu options 
represents the sequence of one or more characters associated with the at least one name menu 
option. 

292. In a menu interface including: 

(1) a menu including a plurality of menu options; and 

(2) a display area with respect to which may be at least partially delimited a plurality of 
selectable regions, each selectable region associated respectively with one of the 
plurality of menu options; 

an apparatus for selecting one of the menu options, said apparatus comprising: 

(a) delimit means for at least partially delimiting the selectable regions; 

(b) indication means, coupled to the delimit means, for displaying each menu option on 
the display area, wherein the location of each displayed menu option indicates the 
location of the associated selectable region and wherein none of the displayed menu 
options intersects the selectable region associated therewith; and 

(c) selection means, responsive to a selection event associated with any one of the 
selectable regions, for selecting the menu option associated with the selectable region 
associated with the selection event. 

293. The apparatus of claim 292 wherein none of the displayed menu options is located adjacent 
the selectable region associated therewith. 

294. The apparatus of claim 293 wherein the relative location of each displayed menu option with 
respect to the other displayed menu options indicates the relative location of the associated 
selectable region with respect to the other selectable regions. 

295. The apparatus of claim 292 wherein each selectable region is located outside the display 
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area. 

296. The apparatus of claim 295 wherein each selectable region is unbounded on at least one side. 

297. The apparatus of claim 292 wherein each selectable region is located on the display area 
adjacent an edge of the display area. 

298. The apparatus of claim 292 wherein the plurality of selectable regions together at least 
partially circumscribe a region on the display area. 

299. The apparatus of claim 292 wherein the plurality of selectable regions together at least 
partially circumscribe a transparent region within the display area. 

300. The apparatus of claim 292 wherein each of the menu options represents a sequence of one 
or more graphic symbols, one or more sequences of the plurality of sequences including one 
or more ideographs, each of the plurality of sequences having a common characteristic. 

30 1 . The apparatus of claim 292 wherein each of the menu options represents a symbol in a 
symbol set. 

302 The apparatus of claim 30 1 wherein the symbol set is any one of the Picture Communication 
Symbols symbol set, Rebus symbol set, Picsyms symbol set, Pictogram Ideogram 
Communication Symbols symbol set, Yerkish symbol set, Blissymbolics symbol set, and a 
set of depictions of the signs of a manual sign language. 

303. The apparatus of claim 292 wherein each of the menu options represents a word, each word 
beginning with a common sequence of one or more characters. 

304. The apparatus of claim 303 wherein the common sequence of one or more characters 
includes a letter from one of the Afrikaans, Albanian, Amharic, Arabic, Armenian, 
Assamese, Assyrian, Avar, Azerbaijani, Balinese, Bamara, Bantu, Bashkir, Basque, Bengali, 
Birhari, Bulgarian, Buluba-Lulua, Burmese. Buryat, Byelorussian, Caddoan, Catalan, 
Chechen, Chikaranga, Chippewa. Choctaw, Church Slavik, Chuvash, Coptic, Cree, Croatian, 
Cyrillic, Czech, Dakota, Danish, Dari, Devanagari, Dutch, Dzongkha, English, Eskimo, 
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Esperanto, Estonian, Ewe, Farsi, Fijian, Filipino, Finnish, Flemish, French, Fulani, Gaelic, 
Galician, Georgian, German, Greek, Gujarati, Gurmakhi, Harari, Hausa, Hawaiian, Hebrew, 
Hindi, Hiragana, Ibo, Icelandic, Indonesian, Irish, Irogquoian, Italian, Kabardian, Kalmyk, 
Kannada, Kanuri, Kashmiri, Katakana. Kazakh. Khasi, Khmer, Kirghiz, Kishmiri, Komi, 

5 Kongo, Kurdish, Lao, Latin, Latvian, Lithuanian, Lu-Guanda, Macedonian, Magahi, 

Maithili, Makua, Malagasy, Malay, Malayalam. Maltese, Mandingo, Manipuri, Marathi, 
Masai, Mizo. Moldavian. Mongolian. Munda, Naga, Navaho, Nyanja, Nepalese, Norwegian, 
Oriya, Oromo, Ossetian, Pashto, Polish, Portugese, Punjabi, Rajasthani, Rhaeto-Romanic, 
Rumanian, Russian, Samoan, Sangs, Serbian, Serbo-Croatian, Sinhalese. Sinhi. Sioux, 

10 Slovak, Slovenia, Spanish, Sundanese, Swahili, Swedish, Syriac, Tadzhik, Tagalog,Tajik, 

Tamil, Tatar, Telugu, Thai, Tibetan, Turkish, Turkmen, Udmurt, Uighur, Ukranian, 
Umbundu, Urdu, Uzbek, Vietnamese, Visayan, Welsh, Yakut, Yoruba and phonetic 
alphabets. 

15 305. The apparatus of claim 292 wherein each of the menu options represents a sequence 

including one or more characters, each sequence including a common sequence of one or 
more characters. 

306. The apparatus of claim 292 wherein the indication means further comprises display means 
20 for displaying a plurality of indicating regions, each indicating region associated respectively 

with one of the menu options, each indicating region displayed in a location intersecting or 
near the associated displayed menu option, and wherein no one indicating region intersects 
the selectable region associated with the menu option associated with said one indicating 
region. 

25 

307. The apparatus of claim 306 wherein at least part of each indicating region is displayed in one 
manner and at least part of the selectable region associated with the menu option associated 
with each indicating region is displayed in a manner similar to the one manner. 

30 308. The apparatus of claim 307 wherein the manner of display is any one of: 

(1) a size; 

(2) a shape; 

(3) a hue; 

(4) a brightness; 
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(5) 
(6) 



a contrast; 



a tone; 



5 



(11) 



(8) 
(9) 
(10) 



(7) 



a dithering; 
a fill; 
a font; 
a blinking; 
a hatching; and 



(12) 



a pattern. 



10 309. The apparatus of claim 306 further comprising movement related signal receiving means for 



receiving a movement related signal indicating a location with respect to the display area; 
and wherein the indication means further comprises intersection indication means for 
indicating an intersection of the location and any one of the selectable regions in the 
indicating region associated with the menu option associated with the intersected selectable 



310. The apparatus of claim 306 further comprising movement related signal receiving means for 
receiving a movement related signal indicating successive locations with respect to the 
display area; and wherein the indication means further comprises intersection indication 
means for indicating the duration of a period of intersection of at least two of the successive 
locations and any one of the selectable regions in the indicating region associated with the 
menu option associated with the intersected selectable region. 

311. The apparatus of claim 306 wherein the selection means further comprises selection 
indication means, responsive to the selection event, for indicating that the selection event has 



3 12. The apparatus of claim 292 wherein the selection means includes a processor coupled to a 
switch and the selection event includes an operation of the switch. 



313. The apparatus of claim 292 wherein the selection means includes timing circuitry and the 
selection event includes a signal from the timing circuitry indicating an expiration of a 
predetermined period of time. 



15 



region. 



occurred. 



30 
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314. A menu option selector comprising: 

(a) a first menu including a first plurality of menu options; 

(b) a surface including a display area with respect to which may be at least partially 
delimited a plurality of selectable regions, each selectable region capable of being 
associated respectively with one of the first plurality of menu options; 

(c) a first plurality of indicators on the display area, each of the first plurality of indicators 
associated respectively with one of the first plurality of menu options, and each one of 
the first plurality of indicators indicating the location of the selectable region capable 
of being associated with the menu option associated with said one of the first plurality 
of indicators; and 

(d) means for selecting any one of the selectable regions. 

315. The menu option selector of claim 314 wherein each of the selectable regions is outside the 
display area. 

316. The menu option selector of claim 315 further comprising a second menu including a second 
plurality of menu options, each of the second plurality of menu options capable of being 
associated respectively with one of the selectable regions; and a second plurality of 
indicators on the display area, each of the second plurality of indicators associated 
respectively with one of the second plurality of menu options, and each one of the second 
plurality of indicators indicating the location of the selectable region capable of being 
associated with the menu option associated with said one of the second plurality of 
indicators. 

317. The menu option selector of claim 316 further comprising association means for associating 
the plurality of selectable regions with either one of the first plurality of menu options or the 
second plurality of menu options. 

318. The menu option selector of claim 3 1 7 wherein the association means is responsive to a 
selection event. 

319. The menu option selector of claim 3 1 7 wherein the association means is responsive to an 
expiration of a predetermined period of time. 
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320. For use with a general purpose computer system including a display with respect to which 
may be at least partially delimited a plurality of selectable regions, each selectable region 
associated respectively with a menu option, an apparatus for selecting a menu option from 
the plurality of menu options, said apparatus comprising: 

(a) a medium readable by the general purpose computer system; and 

(b) a program, stored on the medium and executable by the general purpose computer 
system, for: 

( 1 ) at least partially delimiting the selectable regions with respect to the display; 

(2) displaying each menu option on the display, the location of each displayed menu 
option indicating the location of the associated selectable region and none of the 
displayed menu options intersecting the selectable region associated therewith; 
and 

(3) responsive to a selection event associated with any one of the selectable regions, 
selecting the menu option associated with the selectable region associated with 
the selection event. 

321. In a computer system including a display with respect to which may be at least partially 
delimited a plurality of selectable regions, each selectable region associated respectively with 
a menu option, an apparatus for selecting a menu option from the plurality of menu options, 
said apparatus comprising: 

(a) the computer system; and 

(b) a program, for execution on the computer system, for: 

( 1 ) displaying a plurality of indicating regions on the display, each indicating region 
associated respectively with one of the menu options: 

(2) displaying each menu option in a location intersecting or near the associated 
indicating region, wherein each indicating region: 

(i) indicates the location of the selectable region associated with the menu 
option associated with the indicating region; and 

(ii) does not intersect the selectable region associated with the menu option 
associated with the indicating region; and 

(3) selecting, responsive to a selection event associated with any one of the 
selectable regions, the menu option associated with the selectable region 
associated with the selection event. 
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322. A speech synthesis system comprising: 

(a) a display; 

(b) a speech synthesizer; and 

(c) control means for: 

( 1 ) receiving a movement related signal indicating successive locations on the 
display; 

(2) displaying: 

(i) a plurality of selectable regions on the display, the plurality of selectable 
regions together at least partially circumscribing a region on the display, 
each selectable region associated respectively with a sequence of one or 
more characters; and 

(ii) the plurality of sequences in the at least partially circumscribed region, 
wherein the location of each sequence indicates the location of the 
associated selectable region; 

(2) repetitively, in response to an intersection of one of the successive locations and 
a selectable region and a dwell event, appending the sequence associated with 
the intersected selectable region to at least one previously selected sequence; and 

(3) speaking, by means of the speech synthesizer, the word spelled by the appended 
sequences. 

323. For use with a menu interface including a display area and a menu hierarchy including a 
menu including a plurality of menu options, at least one of the menu options associated with 
a submenu including a plurality of submenu options, a method of selecting a submenu option 
from the menu hierarchy, said method comprising the steps of: 

at least partially delimited a plurality of selectable regions with respect to the display area, 
each selectable region associated respectively with one of the menu options and with one of 
the submenu options; 

displaying a plurality of submenu indicating regions on the display area, each submenu 
indicating region associated respectively with one of the submenu options, wherein each 
submenu indicating region is displayed in a manner indicating the location of the selectable 
region associated with the submenu option associated with the submenu indicating region; 



183 



WO 96/30822 



PCT/US95/03591 



selecting, responsive to a first selection event associated with any one of the selectable 
regions, the menu option associated with the selectable region associated with the first 
selection event; and 

thereafter selecting, responsive to a second selection event associated with any one of the 
selectable regions, the submenu option associated with the selectable region associated with 
the second selection event. 

324. The method of claim 323 further comprising the step of displaying a plurality of menu 
indicating regions, each menu indicating region associated respectively with one of the menu 
options, wherein each menu indicating region is displayed in a manner indicating the location 
of the selectable region associated with the menu option associated with the menu indicating 
region. 

325. The method of claim 324 further comprising the step of displaying a plurality of submenu 
indicating regions, each submenu indicating region associated respectively with one of the 
submenu options, wherein each submenu indicating region is displayed in a manner 
indicating the location of the selectable region associated with the submenu option associated 
with the submenu indicating region; and wherein the submenu indicating regions each 
intersect the menu indicating region associated with the menu option associated with the 
submenu. 



184 



WO 96/30822 



PCI7US95/03S91 



1/67 



■H 

o 

I — I 

Ll 



CJi Cl 
Cl 3 



en c 
Cl Q 



a 



u 

CO 

a 



E 
I 



C 

LU 



CM 



CO 

c 



CL 



c: 
UJ 



CO 



c 



oo 



GE 



00 



LO 



CD 



CM 



CO 



C\J 



C3 



J 



o 

CO 



CO 






Cl 










'sz 


O 






J 







CD 

O 

Q. 

CO 



WO96/30S22 



PCT/US95/03S91 



2/67 



o 

CM 

o 



o 

CNJ 

o 



a 

li 



ID 



>l 



SI 

o 
CO 



3 



a) 



c 

(D 



0 



CD 

c 
n 



o 



o 



CD 
0) 
■D 
O 

.c 

^ "D 

=3 CD 

O "Z 

8* 



C 
O 

o 



oo 



> 

i_ 

CD 

E 
E 

D 

V) 

V) 
D 

o 
o 



CD 
"O 
(0 



z 2 



Q 



o 



(A 
"O 
D 
O 

O 
CD 



c 

03 
CD 
U 
O 

CD 



(0 »4_ 

-F. o 



E 
o 

o 
a 

0 
CD 
"D 



(0 CD 
C 

< £ 



o> a 



cm 



a 

Ul 

m 



CDGG 
CD 



OB 
CDS 

ma 



□DCD 
COB 



a c 
a o 



•o 
c 
Ul 



_Q 



tt) 



LiU 



l- 



/a 



CD 



to 
c 

W 



0> 



CO 



(7) 



0 



if) 

a 

(0 

0 



r 
(/) 



WO 96/30122 



PCT/US9S/03S91 




WO 96/30822 



PCT/US95/03591 




WO96/30S22 



PCT/US9S/03591 



5/67 




o 



LO 

o 

I — I 
LL 



ID 



WO96/30S22 



PCT/DS9S/03S91 



6/67 



Q 
(/) 

CO 



•H 

QJ 



D 
0 



O 

I 1 

Ll 



■o 

UJ 



CD 

CD 
>| 



o 

1 

CO 
C0| 



Si 

CD 



C 

a> 

c 

o 

o 

CO 



o 
o 

-O 
CD 

c 

O 



o 


^ 




CD 


o 


E 




E 


CD 




-t— ■ 


CO 


C 


CO 


> 


Z3 


CD 


O 






o 


GO 


a> 




CD 


$ 


"O 


o 




z: 





C 

_ CD 

o 2 

CD CD 

CO 
CD 

E 



CD 
.Q 
O 



CO 00 

" u 



CO 

=> 

_o 

o 



? CD 



E 

o 

CO 

O 
.Q 

Q. 
CD 
CD 
T5 



CO CD 
< .E 



0 



0 



cr 



N 



0> 



+ 



0) 

0 

Q 
(/) 



CO 



10 



(1) 



CM 



10 



, 0) 



WO 96/30*22 



PCT/US9M0591 



7/67 



C\J 
O 





WO 96/30822 



PCT/US95/D3591 



8/67 




WO 9630822 



PCT/US9S/03591 



11/67 
FIG. 11 



Meaning 


Communication 
Symbols 


Rebus 


want 




f% 


eat 


9r 


<$tr 


think 






where 


* 




friend 


A A 




wheelchair 






television 






yesterday 
no 


.©- no 




not,un 


X 






.1 

000 -ii 

X 




-I 



WO 96/30(22 



PCT/DS95/03591 



12/67 



FIG. 12(a) 



go 



FIG. 12(c) 




wagon 



FIG. 12(b) 




((111 



garden 



FIG. 12(d) 



sad 



SUBSTITUTE SHEET (RULE 26) 



WO 96/30822 



VCT/US95/D3591 



13/67 



FIG. 13(a) 



FIG. 13(b) 








■ 


m 


1 













pretzel 



potato stick 



FIG. 13(c) 





m & m 



tidbit 



FIG. 13(e) 



gumdrop 



FIG. 13(f) 



FIG. 13(g) 



cereal 



peanut 



FIG. 13(h) 







□ 








4 

V 



fizzie 



FIG. 13(i> 




butterscotch 




SUBSTITUTE SHEET (RULE 26) 



WO 9630822 



PC17US9S/03S91 




WO 96/30822 



PCT/US9S/D3591 



15/67 



i 1 




WO 96/30822 



PCT/US9S/03591 



17/67 



CM 




WO 96/30822 



PCT/US95/03591 



18/67 



CM 




WO 96/30*22 



FCT/US95/03S91 



19/67 




WO 96/30822 



. PCT/US95/Q3991 



20/67 




WO 96/30822 



PCT/US95/03591 



21/67 




WO 96/30822 



PCT/US95/03591 



22/67 




WO 96/30822 



PCT/US95/03591 



23/67 




WO 96/30822 



PCT/US95/03591 




WO 96/30822 



PCIYUS95/03591 




WO 96/30822 



PCT/US95/03591 




WO 96/30822 



27/67 



PCT/US95/03591 




O 
ID 
00 
O 
I 

5 

\ 




WO 96/30822 



PCT/US95/03591 



28/67 




WO 96/30822 



PCT/US95/03591 



29/67 





FIG. 29 






Event 






0 


1 


2 


3 


4 


5 


6 


7 


8 


9 






E 


E 


E 


E 


E 


E 


E 


E 


'E 


E 






V 


V 


V 


V 


V 


V 


V 


V 


V 


V 






N 


R 


D 


D 


C 


S 


S 


M 


I 


C 






IT 

KJ 




E 


w 


D 

JV 


T 


T 


n 


r\ 

YJ 


F 






L 


S 


C 


E 


o 


E 


E 


V 


L 


I 






L 


E 


A 


L 


s 


P 


P 


E 


E 


L 








T 


Y 


L 


s 

o 

u 

T 


D 
O 
W 
N 


U 
P 


M 
E 
N 
T 


f 
I 
M 
E 
%j 
U 


I 
N 
G 




State 


















T 




A 
0 


ST_ILLEGAL_STATE 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 


ST_ERROR 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


2 


ST_INITIAL 


0 


3. 


2 


2 


2 


2 


2 


2 


2 


0 


3 


ST_RESET 


0 


3 


3 


3 


12 


0 


0 


12 


3 


0 


4 


ST LOW TIDE 


4 


3 


8 


9 


4 


0 


5 


8 


10 


0 


< 

5 


ST_SELECTED 


6 


3 


0 


0 


0 


0 


0 


0 


0 


0 


6 


STCRESTTIDE 


6 


3 


8 


9 


7 


4 


0 


8 


10 


9 


7 


STSELECTANDOUT 


0 


3 


0 


0 


0 


0 


0 


0 


0 


0 


8 


ST_DECAY 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


9 


ST_DWELL 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 



WO 96/30822 



PCT/US95/03591 



30/67 





FIG. 30 






Event 






0 


1 


2 


3 


4 


5 


6 


7 


8 


9 






E 


E 


E 


E 


E 


E 


E 


E 


E 


E 






V 


V 


V 


V 


V 


V 


V 


V 


V 


V 






N 


R 


D 


D 


C 


S 


S 


M 


I 


C 






¥T 

u 






W 


K 


T 
1 


T 
1 


\j 


V 


IT 






L 


s 


c 


E 


o 


E 


E 


V 


L 


I 






L 


E 


A 


L 


s 


P 


P 


E 


E 


L 








T 


v 

M 


L 


s 

6 
u 

T 


D 

O 
W 
N 


U 
P 


M 
E 
N 
T 


f 
I 
M 
E 
0 
U 


I 

N 
G 




State 


















T 




in 

1U 


ST_IDLE 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 


11 


ST_SELECTED_SEQ_ON 


0 


3 


0 


0 


0 


0 


0 


0 


0 


0 


1Z 


STEBBTIDE 


12 


3 


8 


13 


12 


0 


5 


B 


10 


0 




ST_ENTRY 


4 


0 


0 


0 


0 


0 


0 


0 


0 


0 


14 


ST_BEGIN_TO_LOCK 


0 


3 


15 


15 


15 


0 


0 


15 


10 


15 


15 


ST_LOCK 


16 


3 


15 


15 


15 


0 


0 


15 


10 


15 


16 


STENDLOCK 


0 


3 


17 


17 


7 


0 


0 


17 


10 


17 


17 


ST_DISCARD 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 



WO 96/30822 PCT/US95/03591 




WO 96/30*22 



PCT/US95S03591 




WO 9630822 



PCT/US95/03S9J 



33/67 




WO 9630822 



PCI7US95/03591 




WO 96/30122 



PCT/US95/03591 




WO 9630822 



PCT/US95/03591 



36/67 




WO 96/30822 



PCT/US9&03991 



37/67 



iiiiiii)ii)M)i)M)n 




O 
CO 



WO96/30S22 



PCT/US9S/03591 



38/67 




WO96/30S22 



PCT/US95/03591 




WO 96/30822 



PCT/US93/03591 




WO 96/30*22 



PCT/US95/D3591 



41/67 , 

^ ^ ^ 

CM rj- rr 




WO 96/30822 



FCTSUS9S/03591 




"vf T T CM 

C\J CM CM 



WO 96/30822 



PCT/US95/03591 




WO 96/30822 



PCI7US9MBS91 



44/67 




WO 96/30822 



PCI7US95/03591 



45/67 




WO 96/30*22 



PCIYUS95/D3591 



46/67 




WO 96/30822 



PCT/US95/03591 



47/67 




WO 9630822 



PCT/US95/D3591 



48/67 




WO 96/30822 



PCT/US95/03591 



49/67 




WO 96/30822 



PCTAJS9S/03591 



50/67 




WO 96/30*22 



PCT/US9MJ3591 



51/67 





CvJ 

o3 



WO 96/30822 



PCT/USM/03591 



52/67 




WO 96130*22 



PCT7US95/03591 



53/67 



CM 




00 
CM 



WO 96/30822 



PCT/US95/03S91 




WO 96/30822 



PCT/US95/03S91 



55/67 





WO 96/30*22 



PCI7US95/03591 




WO 9630122 



PCIYUS9S/03591 




WO 96/30122 



PCTAJS95/03591 



58/67 

o 

00 
lO 




C\J 



WO 96/30822 



PCI7US95/03591 




WO96/30S22 



FCT/US95/D3S91 



60/67 




WO 96/30822 



PCT/US95/0359 1 



61/67 




WO96/30S22 



FCT/US95/03S91 




WO 96730*22 



PCT/US9MBS91 





O O 

5 Q <x> 



WO 960001 



PCT/DS9V03591 



64/67 



cm oo o 

oo / oo / 00 / 00 




WO 96/30822 



PCT/US95/03591 



65/67 



00 J LO 

oo / oo / 00 




WO 96/30822 



PCT/US9S/03S91 




WO 96/30122 



PCI7US9S/03591 



67/67 



lO 




INTERNATIONAL SEARCH REPORT 



Inte nil Application No 

PCT/US 95/03591 



According to International Patent Classification (IPC) or lo both national classification and IPC 



pr5 S,F,C G0^3/or WECT G0TF3/033 G06F3/O23 GO9B21/O0 A51F4/00 



B. FIELDS SEARCHED 



Minimum documentation searched (classification system followed by classification symbols) 

PC 6 G06F G09B A61F 



Documentation searched other than minimum documentation to the extent that such documents are included in the fields searched 



Electronic data base consulted during the international search (name of data base and, where practical, search terms used) 



C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 1 Citation of document, with in* canon, where appropriate, of the relevant passages 



Relevant to claim No. 



DE.U.93 00 231 (HASABI I.) 19 August 1993 
see page 1, line 1 - page 6, line 2 



PATENT ABSTRACTS OF JAPAN 

vol. 018 no. 460 (P-1793) ,26 August 1994 

& JP.A.06 149534 (TOSHIBA CORP) 27 May 

1994, 

see abstract 

EP,A,0 324 306 (IBM) 19 July 1989 
see the whole document 



34 

43,44 
34 



31 

1,10,19, 
21,22 



m 



Further documents are listed in the continuation of box C. 



0 



Patent family members are listed in annex. 



' Speaal categories of cited documents : 

'A* document defining the general stale of the art which is not 

considered to be or particular relevance 
"E" earlier document but published on or after the international 

filing date 

"L" document which may throw doubts on priority daimfs) or 
which is ated to establish the publication date of another 
citation or other speaal reason (as specified) 

'O* document referring to an oral disclosure, use, exhibition or 
other means 

'P* document published pnor to the international filing date but 
later than the priority date claimed 



T later document published after the international filing date 
or priority date and not in conflict with the application but 
ated to understand the principle or theory underlying the 
invention 

'X' document of particular relevance; the claimed invention 
cannot be considered novel or cannot be considered to 
involve an inventive step when the document is taken alone 

'Y' document of particular relevance, the claimed invention 
cannot be considered to involve an inventive step when the 
document is combined with one or more other such docu- 
ments, such combination being obvious to a person skilled 
in the art. 

document member of the same patent family 



Date of the actual completion of the international search 

12 March 1996 



Name and mailing address of the ISA 

European Patent Office, P.B. 5818 Patendaan 2 
NL - 2280 HV Rijswi]k 
Tel. ( + 11-70) 340-2040, Tx. 31 651 epo nl, 
Fax: ( + 31-70)340-3016 



Date of mailing of the international search report 



28.03.96 



Authorized officer 



Bailas, A 



Foim PCT ISA/210 (ncond shall) (July IW2) 



page 1 of 3 



INTERNATIONAL SEARCH REPORT 



Inter rial Application No 

PCT/US 95/03591 



C(Conanuanon) DOCUMENTS CONSIDERED TO BE RELEVANT 


Category " 


Citation of document, with indication, where appropriate, of the relevant passages 


Relevant to claim No. 


V 
T 


Ui,A,4 lUy 143 IbKAr LAKL r) LL August 


31 




iy/8 






see column 2, line 39 - line 59 






see column 3, line 13 - column 8, line 30 




A 

A 




1 "5 c ~r 

1,3,6,7, 






1ft or 

19-25, 






30,130, 






164,165, 






228,322 


A 

A 


bD,A,£ 1/i Uzij (INTtlH bYbltMS INC) 1 


1 o c ft 

1,2,5,8, 




uctoDer l9oo 


1 A 1 ^ 1 ft 

14,17,19 




caa nsnA O 1 i ma CO nasi a /I 1 • n — , ft 

see page c, line ■ Page i, line y 




A 

A 


tUMru 1 tK, 


1 A IE 

1,4,15, 




, . l o£ -> t i /-\ c ai a kj t Tr\r iir 

vol. 26, no. 7, LOS ALAM1T0S US, 


1 ft ftft 

19,20 




pages 65-66, XP 000376643 




JACOB R. J. K. ET AL 'What you look is 






what you get' 






see page 65, left column, line 1 - middle 






column, line 12 






see page 66, middle column, line 6 - line 






63 




A 


IBM TECHNICAL DISCLOSURE BULLETIN, 


28,29,32 




, . _ 1 -\ o _ ou aVi r~i f unni/ lie 

vol. 32, no. 8D, NEW YORK US, 




pages 231-232, XP 000082441 PRE - 






en r^Tt ah 1 1 t ft 1 1 ■ t n iti ur i 

SELECTION HIGHLIGHTING" 






see the whole document 




A 


IBM TECHNICAL DISCLOSURE BULLETIN, 


41,43, 




vol. 33, no. 10A, 1 March 1991 


A A A af 

44,46 




fisnA 070 /no vd nnni innoo intcrnrTr Aim 

page i/o/j/y xp oooiiooyt: discrete and 






lUNIlNUUUi IUNIKUL UolNu olAHLt brttLH 












see tne wnoie document 




A 


WO, A, 86 03863 (UNIV ADELAIDE ;D0WNING 


130 




ANDREW RICHARD (AUJ ) 3 July 1986 






see page 1, line 1 - page 3, line 3 






see page 4, line lb - line 22 






see page 5, line 6 - line 33 






m A A * ft- A O 1 i a aC n A O 1 i n m 1 1 

see page o, line o - page y, line 11 




A 

A 


ire DDnrccm wee a duvctta! crTciurr 
let rKULttUl Nu3 A rniilLAL OLitNLt, 


3c, 3/ , 




MCACIIDCMCMT 9. T MCTDIIMCWTATT I"1H MAMATCMCMT 

ntAbUKtHtNl & 1N3 1 KUHtN 1 A 1 1UN , MANAutntNl 






& tuULAl 1UN. , 






vol. 134, no. 4, April 1987 STEVENAGE GB, 






pages 328-330, 






YAMADA M. ET AL 'Eye word processor (EWP) 






and peripheral controller for the ALS 






patient' 






see page 329, right column, line 1 - line 






21; figure 5 






-/-- 





i PCT.1SA/310 (centinuiion of • 



I thmi) (July l*M) 



page 2 of 3 



INTERNATIONAL SEARCH REPORT 



Intr rial Application No 

PCr/US 95/03591 



C<Cononuanon) DOCUMENTS CONSIDERED TO BE RELEVANT 



Category ' 


Citation of document, with indication, where appropriate, of the relevant passages 


Relevant to claim No. 


A 


US,A,4 788 649 (SHEA F WILLIAM ET AL) 29 
November 1988 

see column 19, line 1 - line 25 


125,130, 
164,165, 
228,322 



Form PCT ISA/210 (continuation of tncond thail) (July 



page 3 of 3 



INTERNATIONAL SEARCH REPORT 



•matronal application No. 

PCT/US 95/ 03591 



Box 1 Observations where certain claims were found unsearchable (Continuation of item I of first sheet) 

This iniernauonal search report has not been established in respect of certain claims under Article 1 7(2)(a) for the following reasons: 
1. [^] Claims Nos.: 

because they relate to subject mailer not required to be searched by this Authority, namely: 



2. Claims Nos.: 

because they relate lo parts of ihe international application thai do not comply with the prescribed requirements to such 
an extent lhat no meaningful iniernauonal search can be carried out, specifically: 



3. Q Claims Nos.: 

because they are dependent claims and are not drafted in accordance with the second and third sentences of Rule 6.4(a). 

Box II Observations where unity of invention is lacking (Continuation of item 2 of first sheet) 
This Iniernauonal Searching Authority found multiple inventions in ihis international application, as follows: 

SEE ANNEXED SHEET 



As all required additional search fees were timely paid by ihe applicant, this international search report covers all 
searchable claims. 

As all searchable claims could be searches without effort justifying an additional fee, this Authority did not invite pavment 
of any additional fee. 



3. _X_ As only some of the required additional search fees were timely paid by the applicant, this iniernauonal search report 
covers only those claims for which fees were paid, specifically claims Nos.: 

1-32, 33-57, 125, 130, 164-165, 228, 252, 289, 322 



4. | | No required additional search fees were umely paid by the applicant. Consequently, this iniernauonal search report is 
restricted to the invention first mentioned in the claims; il is covered by claims Nos.: 



Remark on Protest 




The additional search fees were accompanied by the applicant s protest. 



I ~J No protest accompanied ihe payment of additional search fees. 



Form PCT.'ISA.llO (continuation of first sheet (1)) (July 1992) 



INTERNATIONAL SEARCH REPORT 

INVITATION TO PAY ADDITIONAL FEES 



International application No. 

PCT/ US 95/ 0359 1 



1. Claims 1-32: Method of selecting a sequence of graphic symbols 
(ideographs). 

2. Claims 33-57, 125, 130, 164-165, 228, 252, 289, 322: Speech 
synthesizer and menu selection by sound. 

3. Claims 58-121, 128, 154-155, 173-174, 180-221, 224-227, 233-236, 
238-251, 253, 292-321, 323-325: Menu selection methods. 

4. Claims 122-124, 131, 159-160, 163, 166, 175-176, 222, 281-283, 290- 
291 : Character and word data entry method. 

5. Claims 132-153, 177-179: Indicating cursor dwell time at a specified 
position. 

6. Claims 167-172, 275-280: Ordering system. 

7. Claims 254-174: Cursor alignment/movement method. 

8. Claims 230-231 , 237: Method for measuring disabled person's ability to 
move a limb. 

9. Claims 126-127, 156-158, 229-230: Cursor controller. 

The special technical feature of the first invention is: 

A method of selecting a sequence of graphic symbols (ideographs). 

According to Rule 13 PCT non-unity a priori is detected for inventions 2-9 
which deal with the following: 

2. Speech synthesizer and menu selection by sound. 

3. Menu selection methods. 

4. Character and word data entry method. 

5. Indicating cursor dwell time at a specified position. 

6. Ordering system. 

7. Curser alignment/movement method. 

8. Method to measure disabled person's ability to move a limb. 

9. Cursor controller. 



Form PCT,ISA,206 (extra sheet) (July 1992) 



INTERNATIONAL SEARCH REPORT 

iniormaoon on patent lamiiy manners 



Inte mal Application No 

PCT/US 95/03591 



Patent document 
cited in search report 


Publication 
dale 


Patent family 
member(s) 


Publication 
date 


DE-U-9300231 


19-08-93 


NONE 






EP-A-0324306 


19-07-89 


US-H- 
JP-A- 


H716 
1136227 


05-12-89 
29-05-89 


US-A-4109145 


22-08-78 


N0NE 






GB-A-2173023 


01-10-86 


US-A- 
CA-A- 
JP-A- 
NL-A- 


4698758 
1235818 
61234465 
8502825 


06-10-87 
26-04-88 

1 n i a tie 

18-10-86 
16-10-86 


WO-A-8603863 


03-07-86 


AU-B- 
GB-A- 


5237786 
2179147 


22-07-86 
25-02-87 


US-A-4788649 


29-11-88 


N0NE 







Form PCT/IS A/310 



■> (July l»M) 



